<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
    <title>Andrei Kazialetski — AVKSoft</title>
    <subtitle>Software engineering, product development, and technical writing by Andrei Kazialetski.</subtitle>
    <link rel="self" type="application/atom+xml" href="https://iamavk.info/atom.xml"/>
    <link rel="alternate" type="text/html" href="https://iamavk.info"/>
    <generator uri="https://www.getzola.org/">Zola</generator>
    <updated>2026-12-18T00:00:00+00:00</updated>
    <id>https://iamavk.info/atom.xml</id>
    <entry xml:lang="en">
        <title>Testing AI Agents With Independent Agents</title>
        <published>2026-12-18T00:00:00+00:00</published>
        <updated>2026-12-18T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/independent-agent-testing/"/>
        <id>https://iamavk.info/blog/independent-agent-testing/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/independent-agent-testing/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/independent-agent-testing/hero.png&quot; alt=&quot;Abstract illustration of independent agents following separate paths through shared evidence&quot; /&gt;&lt;/p&gt;
&lt;p&gt;“Ask another agent to review it” is becoming a standard answer to the reliability problem. Sometimes it helps. Sometimes it produces several confident copies of the same mistake.&lt;/p&gt;
&lt;p&gt;The difference is methodology.&lt;/p&gt;
&lt;p&gt;In the research behind this series, independent agents included Codex, Claude, OpenCode, and local-model review runs. They were useful not because a particular brand is automatically objective, but because separate agents can expose different failure boundaries—if the experiment prevents them from sharing an unexamined conclusion.&lt;/p&gt;
&lt;p&gt;Independence is not a label. It is a property of the path from input evidence to output judgment.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/independent-agent-testing/independent-agents.svg&quot; alt=&quot;Independent reviewers following separate paths through the same evidence&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;what-independence-should-mean&quot;&gt;What independence should mean&lt;/h2&gt;
&lt;p&gt;For an agent reviewer, independence has at least four parts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the reviewer receives the same bounded evidence bundle rather than an evolving shared conversation;&lt;/li&gt;
&lt;li&gt;the reviewer has no access to another reviewer’s conclusion before producing its own;&lt;/li&gt;
&lt;li&gt;the reviewer’s tool calls and outputs are recorded separately;&lt;/li&gt;
&lt;li&gt;the scoring rubric is applied after the outputs are collected.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Two agents in the same long-running chat are not independent merely because they have different names. Two agents that both read an earlier reviewer’s summary are not independent evidence of the summary’s correctness.&lt;/p&gt;
&lt;p&gt;The goal is not to create an artificial contest. It is to make disagreements inspectable.&lt;/p&gt;
&lt;h2 id=&quot;a-practical-test-layout&quot;&gt;A practical test layout&lt;/h2&gt;
&lt;p&gt;I use a five-stage layout.&lt;/p&gt;
&lt;h3 id=&quot;1-freeze-the-question&quot;&gt;1. Freeze the question&lt;/h3&gt;
&lt;p&gt;Write the target claims before running reviewers. Give each claim an ID. Do not let the reviewer redefine the question while answering it.&lt;/p&gt;
&lt;p&gt;For the Axum case, the claims included a purported list of removed types, an assurance that no code changes were needed, and an exact &lt;code&gt;TypedHeader&lt;/code&gt; import path. Negative controls were included as well: claims that the supplied evidence directly supported.&lt;/p&gt;
&lt;h3 id=&quot;2-freeze-the-evidence&quot;&gt;2. Freeze the evidence&lt;/h3&gt;
&lt;p&gt;Build a small evidence bundle from primary sources. Preserve source identity, version, excerpt boundaries, and hashes. The reviewer should see the exact text being evaluated—not a citation that it must retrieve again.&lt;/p&gt;
&lt;p&gt;This prevents a reviewer from quietly changing the experiment by finding a newer page, a different version, or a familiar answer from prior knowledge.&lt;/p&gt;
&lt;h3 id=&quot;3-separate-roles&quot;&gt;3. Separate roles&lt;/h3&gt;
&lt;p&gt;One agent can collect evidence. Another can draft. A third can review the draft. But the reviewer should receive the evidence bundle and target answer as explicit inputs, with instructions not to retrieve or rely on prior knowledge.&lt;/p&gt;
&lt;p&gt;In the Context Mode investigation, the agent that explored the source-resolution behavior was not treated as proof that its proposed upstream contract had been accepted. In the ExecutionJournal work, the code review status &lt;code&gt;ACCEPT&lt;/code&gt; was reported as a review result for the test change, not as a universal guarantee about recovery. The later failure-boundary audit reported &lt;code&gt;1,312 passed&lt;/code&gt; across the defined matrix, but that remains evidence about the tested checkout—not a universal reliability guarantee.&lt;/p&gt;
&lt;h3 id=&quot;4-add-negative-and-insufficiency-controls&quot;&gt;4. Add negative and insufficiency controls&lt;/h3&gt;
&lt;p&gt;A useful reviewer must do more than find errors. It must leave supported claims alone and abstain when the evidence is incomplete.&lt;/p&gt;
&lt;p&gt;The Axum fixture included an insufficiency control asking whether the evidence proved the exact &lt;code&gt;TypedHeader&lt;/code&gt; import path. The safe answer was abstention. A reviewer that “corrects” the target by inventing the path has failed even if it detected that the original answer was risky.&lt;/p&gt;
&lt;h3 id=&quot;5-score-the-contract-in-pieces&quot;&gt;5. Score the contract in pieces&lt;/h3&gt;
&lt;p&gt;Record detection, classification, evidence attribution, uncertainty, correction quality, and false-positive controls separately. Keep parse errors, timeouts, and truncated generations outside the reasoning score.&lt;/p&gt;
&lt;p&gt;The current review runner also validates the response envelope before scoring it. Exact finding/control IDs, required fields, non-empty uncertainty, and explicit abstention for insufficiency are part of the test contract. A response that is valid JSON but violates that schema is a protocol failure, not a reasoning score.&lt;/p&gt;
&lt;p&gt;This makes disagreements useful. One agent may detect F1 but call it “unsupported” instead of “contradiction.” Another may classify it correctly but cite the wrong excerpt. Those are different engineering problems.&lt;/p&gt;
&lt;h2 id=&quot;what-multiple-agents-can-reveal&quot;&gt;What multiple agents can reveal&lt;/h2&gt;
&lt;p&gt;Independent agents are particularly useful for finding:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;claims that sound plausible but are not licensed by the evidence;&lt;/li&gt;
&lt;li&gt;source-version confusion;&lt;/li&gt;
&lt;li&gt;overconfident exact API details;&lt;/li&gt;
&lt;li&gt;retrieval strategies that repeatedly return the same text;&lt;/li&gt;
&lt;li&gt;corrections that introduce a new unsupported assertion;&lt;/li&gt;
&lt;li&gt;rubric weaknesses that binary pass/fail scoring hides.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;They are less useful when the task is underspecified, the evidence is mutable, or the runs share hidden state. More outputs then create the illusion of consensus without increasing information.&lt;/p&gt;
&lt;p&gt;Repeated behavior must also be counted honestly. If a local model returns five byte-identical reviews, that is one observed behavior repeated five times. It is not five independent votes.&lt;/p&gt;
&lt;h2 id=&quot;the-limits&quot;&gt;The limits&lt;/h2&gt;
&lt;p&gt;This methodology does not produce a universal reliability percentage. A handful of agents and a single evidence bundle cannot rank models generally. Different agents may also share training data, familiar conventions, or the same blind spot.&lt;/p&gt;
&lt;p&gt;Independence reduces correlated failure; it does not eliminate it.&lt;/p&gt;
&lt;p&gt;Nor does a reviewer replace tests. The crash-recovery article in this series rests on an executable two-process regression, not on an agent saying that recovery “looks correct.” The Context Mode article rests on a real MCP differential reproduction, not on a reviewer’s confidence about an error contract.&lt;/p&gt;
&lt;p&gt;The strongest workflow combines independent judgment with executable or protocol-level checks wherever possible.&lt;/p&gt;
&lt;h2 id=&quot;a-small-operating-checklist&quot;&gt;A small operating checklist&lt;/h2&gt;
&lt;p&gt;Before trusting an independent-agent review, ask:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Did every reviewer see the same immutable evidence?&lt;/li&gt;
&lt;li&gt;Were unsupported claims and negative controls included?&lt;/li&gt;
&lt;li&gt;Were the reviewers prevented from seeing each other’s conclusions?&lt;/li&gt;
&lt;li&gt;Were retrieval, parsing, and reasoning failures separated?&lt;/li&gt;
&lt;li&gt;Can every correction be traced to the evidence, or does it abstain?&lt;/li&gt;
&lt;li&gt;Were duplicate outputs counted as repeated behavior rather than independent confirmation?&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;If the answer to these questions is yes, multiple agents can be a powerful diagnostic instrument. If not, they may only make an unsupported conclusion sound more social.&lt;/p&gt;
&lt;p&gt;The point of independent agents is not to manufacture consensus. It is to create several auditable paths through the same evidence—and to learn exactly where those paths diverge.&lt;/p&gt;
&lt;p&gt;Related: &lt;a href=&quot;/blog/evidence-to-claim-verification/&quot;&gt;Evidence-to-Claim Verification: Why LLM Reviewers Still Fail&lt;/a&gt;, &lt;a href=&quot;/blog/crash-window-execution-journal/&quot;&gt;The Crash Window&lt;/a&gt;, and &lt;a href=&quot;/blog/a-successful-search-that-never-searched/&quot;&gt;A Successful Search That Never Searched&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Evidence-to-Claim Verification: Why LLM Reviewers Still Fail</title>
        <published>2026-12-11T00:00:00+00:00</published>
        <updated>2026-12-11T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/evidence-to-claim-verification/"/>
        <id>https://iamavk.info/blog/evidence-to-claim-verification/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/evidence-to-claim-verification/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/evidence-to-claim-verification/hero.png&quot; alt=&quot;Abstract illustration of a layered evidence-to-claim verification pipeline&quot; /&gt;&lt;/p&gt;
&lt;p&gt;An LLM reviewer can spot that an answer is suspicious and still produce an unsafe review.&lt;/p&gt;
&lt;p&gt;It may identify the right sentence but assign the wrong error class. It may cite a relevant document without showing that the document supports the claim. It may correctly say that an exact API detail is missing, then invent the detail in its proposed correction.&lt;/p&gt;
&lt;p&gt;That is why “ask another model to review it” is not a complete verification methodology.&lt;/p&gt;
&lt;p&gt;I built a small, bounded review pilot around an Axum migration answer. The pilot compared independent GLM-4.7-Flash and Qwen3.8-27B runs against the same immutable target answer, evidence bundle, and system instruction. The runner made at most five stateless requests, controlled the endpoint parameters, preserved response statuses, validated the review against a strict JSON schema, and did not invoke Agent Loop or retrieval tools.&lt;/p&gt;
&lt;p&gt;The experiment is deliberately modest. It does not establish which model is better in general. It shows why the evaluation contract must score several kinds of correctness separately.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/evidence-to-claim-verification/evidence-review.svg&quot; alt=&quot;Evidence-to-claim verification pipeline&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-findings-and-controls&quot;&gt;The findings and controls&lt;/h2&gt;
&lt;p&gt;The current fixture separates four findings, three negative controls, and one insufficiency control. That distinction matters because a review must identify bad claims without flagging claims that the evidence actually supports.&lt;/p&gt;
&lt;p&gt;F1 described a list of Axum types and traits as removed. The bounded release-note evidence instead described a change involving the request-body generic parameter. This is a contradiction: the wording changes the subject of the release note.&lt;/p&gt;
&lt;p&gt;F2 claimed that no code changes were required because the change was only an internal implementation detail. The evidence did not establish that assurance. The right classification is unsupported, not “supported” and not necessarily “contradiction.”&lt;/p&gt;
&lt;p&gt;F3 supplied the exact line:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;use&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; axum_extra&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;::&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;extract&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;::&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;TypedHeader&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The evidence said that &lt;code&gt;TypedHeader&lt;/code&gt; moved to the &lt;code&gt;axum-extra&lt;/code&gt; crate, but did not establish the exact import path. The correct response is insufficient evidence with abstention—not a guessed replacement.&lt;/p&gt;
&lt;p&gt;F4 asks a narrower question: whether the listed types and traits are no longer generic over the request-body parameter &lt;code&gt;B&lt;/code&gt;. The same evidence that contradicts F1 supports F4. Keeping these claims separate prevents a reviewer from confusing “the APIs were removed” with “their genericity changed.”&lt;/p&gt;
&lt;p&gt;The negative controls cover the &lt;code&gt;axum::serve&lt;/code&gt; example, the &lt;code&gt;TypedHeader&lt;/code&gt; crate move, and the &lt;code&gt;axum::body::Body&lt;/code&gt; replacement. The insufficiency control asks whether the evidence proves the exact &lt;code&gt;TypedHeader&lt;/code&gt; import path; it requires abstention. A reviewer that flags everything, or invents a path for the insufficiency control, is not reliable.&lt;/p&gt;
&lt;h2 id=&quot;five-dimensions-not-one-score&quot;&gt;Five dimensions, not one score&lt;/h2&gt;
&lt;p&gt;The runner kept these dimensions separate:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Detection:&lt;/strong&gt; did the reviewer identify the problematic claim?&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Classification:&lt;/strong&gt; did it distinguish contradiction, unsupported, and insufficient evidence?&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Attribution:&lt;/strong&gt; did it cite the relevant evidence item?&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Uncertainty:&lt;/strong&gt; did it state what the bundle did not establish?&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Correction:&lt;/strong&gt; did the proposed correction stay inside the evidence boundary?&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This separation exposed a common scoring trap. A reviewer can set &lt;code&gt;detected: true&lt;/code&gt; for F3, cite E3, and still fail because its correction invents &lt;code&gt;axum_extra::extract::TypedHeader&lt;/code&gt;. Detection alone measures claim coverage, not safe review.&lt;/p&gt;
&lt;p&gt;Similarly, evidence attribution is not the same as evidence sufficiency. Citing E1 does not make an unsupported assurance true. The citation has to establish the claim being made.&lt;/p&gt;
&lt;h2 id=&quot;the-schema-is-part-of-the-experiment&quot;&gt;The schema is part of the experiment&lt;/h2&gt;
&lt;p&gt;The review envelope is fail-closed. It must contain exactly &lt;code&gt;findings&lt;/code&gt;, &lt;code&gt;negative_controls&lt;/code&gt;, and &lt;code&gt;insufficiency_control&lt;/code&gt;. Findings must use the expected IDs and fields: detection, one of four classifications, evidence IDs, a non-empty uncertainty phrase, a string or null correction, and an abstention flag. Controls must use the expected IDs and evidence lists. &lt;code&gt;I1&lt;/code&gt; must remain &lt;code&gt;insufficient_evidence&lt;/code&gt; with &lt;code&gt;abstain: true&lt;/code&gt; and a non-empty reason.&lt;/p&gt;
&lt;p&gt;The runner accepts a strictly fenced JSON response as a transport representation, then validates the parsed object against the schema. Empty or truncated generations, malformed JSON, transport failures, and schema-invalid reviews are not silently converted into incorrect or correct reasoning scores; they remain incomplete or parse-error outcomes.&lt;/p&gt;
&lt;p&gt;A second pinned Axum follow-up fixture tests a different boundary: three direct contradictions (&lt;code&gt;F1&lt;/code&gt;–&lt;code&gt;F3&lt;/code&gt;), one supported negative control (&lt;code&gt;N1&lt;/code&gt;), and an insufficiency control (&lt;code&gt;I1&lt;/code&gt;) for a complete project-specific migration patch. It uses separately hashed target and evidence inputs and shares the same fail-closed validation discipline. This is a follow-up fixture, not an additional claim about model quality.&lt;/p&gt;
&lt;h2 id=&quot;immutable-inputs-matter&quot;&gt;Immutable inputs matter&lt;/h2&gt;
&lt;p&gt;The fixture builder extracted the canonical final answer and evidence text from a preserved session, then checked SHA-256 hashes before running the pilot. The expected final-answer hash was:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;941e0664e1b26cb13cb06115c0a5fed8d8bc9e8f334ef30141c6250de9208fd9&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The canonical evidence bundle had its own expected hash:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;09f324307c26d108c77f24409131ac864c324dd14be25d710a4ee8c419d265d1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The hashes are not scientific evidence about answer quality. They are experiment hygiene: they make it possible to say that GLM and Qwen saw the same inputs.&lt;/p&gt;
&lt;p&gt;The runner also treated empty or truncated generations as incomplete rather than incorrect reviews. Malformed JSON was a parse error and was not silently repaired for scoring. That distinction prevents transport and formatting failures from being mistaken for reasoning failures.&lt;/p&gt;
&lt;h2 id=&quot;what-the-comparison-actually-establishes&quot;&gt;What the comparison actually establishes&lt;/h2&gt;
&lt;p&gt;The pilot establishes that independent model outputs can differ in useful ways, but also that repeated identical outputs are not independent successes. Five byte-identical GLM outputs should be counted as one repeated behavior, not five separate confirmations. The preserved Qwen synthesis runs add an important limitation: the model consumed the available completion budget as reasoning and returned truncated or empty visible content, so those runs are generation-budget failures, not evidence that Qwen made a particular review judgment.&lt;/p&gt;
&lt;p&gt;It also shows why a qualitative reading is needed alongside the official rubric. A strict scorer may mark evidence attribution false because a model cites the wrong item even when the model recognized the technical problem. Conversely, a model can receive credit for detection while proposing a correction that is unsafe.&lt;/p&gt;
&lt;p&gt;Those observations do not justify ranking GLM or Qwen from a handful of calls. They justify improving the measurement before making such a claim.&lt;/p&gt;
&lt;h2 id=&quot;a-bounded-correction-is-a-feature&quot;&gt;A bounded correction is a feature&lt;/h2&gt;
&lt;p&gt;For evidence review, abstention is not failure. When an evidence bundle says “moved to &lt;code&gt;axum-extra&lt;/code&gt;” but not the exact module path, the safe correction is:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;The evidence establishes the crate move but not the exact import path.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Verify the path in the versioned API documentation before changing code.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That answer is less satisfying than a code snippet, but it is faithful. A reviewer should not trade evidence discipline for apparent completeness.&lt;/p&gt;
&lt;h2 id=&quot;the-lesson-for-agent-systems&quot;&gt;The lesson for agent systems&lt;/h2&gt;
&lt;p&gt;Evidence verification is a pipeline, not a single model call. The system needs immutable inputs, explicit claim IDs, negative controls, parse-status handling, independent runs, and separate scores for detection, attribution, uncertainty, and correction.&lt;/p&gt;
&lt;p&gt;Without those boundaries, “the reviewer found an issue” can conceal several different outcomes. The reviewer may have found the issue, misunderstood it, cited unrelated text, and proposed a new unsupported assertion—all in one response that receives a flattering binary score.&lt;/p&gt;
&lt;p&gt;The useful question is not “did the model review the answer?” It is “which parts of the verification contract did the model satisfy, and which did it violate?”&lt;/p&gt;
&lt;p&gt;Related: &lt;a href=&quot;/blog/more-evidence-less-reliable/&quot;&gt;When More Evidence Makes an AI Agent Less Reliable&lt;/a&gt; and &lt;a href=&quot;/blog/independent-agent-testing/&quot;&gt;Testing AI Agents With Independent Agents&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>When More Evidence Makes an AI Agent Less Reliable</title>
        <published>2026-12-04T00:00:00+00:00</published>
        <updated>2026-12-04T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/more-evidence-less-reliable/"/>
        <id>https://iamavk.info/blog/more-evidence-less-reliable/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/more-evidence-less-reliable/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/more-evidence-less-reliable/hero.png&quot; alt=&quot;Abstract illustration of evidence overload and retrieval redundancy&quot; /&gt;&lt;/p&gt;
&lt;p&gt;More evidence sounds like a safe strategy for an AI agent. If the answer is uncertain, search again. If a migration detail is missing, retrieve another page. If a citation feels weak, collect one more source.&lt;/p&gt;
&lt;p&gt;That instinct is useful until retrieval becomes a substitute for interpretation.&lt;/p&gt;
&lt;p&gt;I examined two Agent Loop sessions answering an Axum 0.6-to-0.7 migration question: a baseline session and a session given stronger evidence-discipline instructions. The investigation reconstructed tool calls, source identities, repeated evidence, and the path from retrieved text to final claims.&lt;/p&gt;
&lt;p&gt;The result is not a claim that “more retrieval always hurts.” It is narrower: additional retrieval did not automatically correct interpretation errors, and some searches repeated or failed to improve the evidence bundle. The final answer could still misstate what the primary source said.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/more-evidence-less-reliable/retrieval-ab.svg&quot; alt=&quot;Retrieval A/B measurements from the preserved Axum sessions&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;two-different-failure-classes&quot;&gt;Two different failure classes&lt;/h2&gt;
&lt;p&gt;The sessions exposed two problems that are easy to mix together.&lt;/p&gt;
&lt;p&gt;The first was retrieval inefficiency: repeated searches, non-indexed source labels, and operations that added no new relevant evidence. That is a tool-use problem.&lt;/p&gt;
&lt;p&gt;The second was evidence interpretation. The final answer described a list of Axum types and traits as removed, while the release evidence was about the removal of a request-body generic parameter. It also claimed that no code changes were required and supplied an exact &lt;code&gt;TypedHeader&lt;/code&gt; import path that the evidence bundle did not establish.&lt;/p&gt;
&lt;p&gt;Those are not the same defect. A perfect retrieval plan cannot make a model faithfully interpret evidence it has already received. Conversely, a careful reviewer cannot recover evidence that the retrieval phase never found.&lt;/p&gt;
&lt;h2 id=&quot;what-the-a-b-comparison-was-for&quot;&gt;What the A/B comparison was for&lt;/h2&gt;
&lt;p&gt;The comparison was designed to keep the question falsifiable. Both sessions were preserved as JSON records. The analysis reconstructed the chronological tool-call sequence rather than judging only the final prose.&lt;/p&gt;
&lt;p&gt;For each search, the investigator asked whether it introduced a new source, new content from a known source, repeated content, no relevant result, or an error. A source URL appearing twice was not treated as proof of redundancy; the returned content had to be compared.&lt;/p&gt;
&lt;p&gt;That distinction matters. A second search on the same release page can retrieve a genuinely different section. Conversely, a different query can return the same excerpt.&lt;/p&gt;
&lt;p&gt;The analysis also separated “the agent searched too much” from “the answer reasoned incorrectly.” That separation is the main value of the experiment.&lt;/p&gt;
&lt;p&gt;The offline analyzer makes the difference measurable for these two preserved sessions. The baseline used 7 retrieval calls, produced 48 recognized source/chunk references, and repeated 24 reference instances across 24 unique pairs. Three calls contained embedded prior output, so their references were flagged rather than counted as proven fresh evidence.&lt;/p&gt;
&lt;p&gt;The evidence-discipline session used 32 retrieval calls and produced 181 recognized references, but only 46 unique source/chunk pairs: 135 reference instances were repeats. It also recorded 5 unindexed-source calls, 4 no-hit calls, and 2 pagination-exhaustion signals. The analyzer found no exact repeated query groups, which is a useful caveat: redundancy was not simply copy-pasted arguments. It arose from overlapping source retrieval, pagination, and source-selection behavior.&lt;/p&gt;
&lt;p&gt;These are measurements of the saved traces and parser signals, not a universal retrieval-efficiency score. The analyzer explicitly does not decide whether a call was semantically necessary.&lt;/p&gt;
&lt;h2 id=&quot;the-axum-evidence-boundary&quot;&gt;The Axum evidence boundary&lt;/h2&gt;
&lt;p&gt;The canonical evidence bundle used later for independent review contained four bounded items:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;release-note text about removed or changed APIs;&lt;/li&gt;
&lt;li&gt;release-note text about the &lt;code&gt;hyper::Body&lt;/code&gt; re-export;&lt;/li&gt;
&lt;li&gt;release-note text saying that &lt;code&gt;TypedHeader&lt;/code&gt; moved to &lt;code&gt;axum-extra&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;a docs.rs example using &lt;code&gt;axum::serve(listener, app).await.unwrap()&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The evidence directly supported the &lt;code&gt;axum::serve&lt;/code&gt; example, the move of &lt;code&gt;TypedHeader&lt;/code&gt; to the &lt;code&gt;axum-extra&lt;/code&gt; crate, and the &lt;code&gt;axum::body::Body&lt;/code&gt; replacement described in the release material. It did not establish the exact import path inside &lt;code&gt;axum-extra&lt;/code&gt;. It also did not support the broader assurance that no code changes were required.&lt;/p&gt;
&lt;p&gt;Most importantly, the release evidence did not support rewriting “the request-body generic parameter B was removed” into “all these types and traits were removed.” That is a semantic difference, not a missing citation.&lt;/p&gt;
&lt;h2 id=&quot;why-additional-evidence-can-reduce-reliability&quot;&gt;Why additional evidence can reduce reliability&lt;/h2&gt;
&lt;p&gt;Every retrieval result adds text that the model must keep distinct. If the runtime does not preserve source boundaries, version labels, and claim-to-evidence links, more material increases the number of ways a statement can be assembled incorrectly.&lt;/p&gt;
&lt;p&gt;The risk is not that a longer context window magically makes the model less intelligent. The risk is operational:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;repeated excerpts look like independent confirmation;&lt;/li&gt;
&lt;li&gt;nearby release-note bullets are merged into one claim;&lt;/li&gt;
&lt;li&gt;a source name is remembered while its exact wording is lost;&lt;/li&gt;
&lt;li&gt;a missing API detail is filled with familiar prior knowledge;&lt;/li&gt;
&lt;li&gt;the final answer cites a source that is relevant but does not establish the exact sentence.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Retrieval therefore needs a stopping rule based on claim coverage, not on a desire to make the evidence pile larger.&lt;/p&gt;
&lt;h2 id=&quot;a-better-stopping-rule&quot;&gt;A better stopping rule&lt;/h2&gt;
&lt;p&gt;For each intended claim, record one of four states:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;SUPPORTED            the evidence states the claim&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;CONTRADICTED         the evidence states something incompatible&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;INSUFFICIENT         the evidence is relevant but incomplete&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;UNVERIFIED           the required source was not obtained&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Stop searching when the claims that matter are classified—or when the next search has a specific hypothesis about which missing state it will resolve. Do not search merely because the answer feels uncomfortable.&lt;/p&gt;
&lt;p&gt;That rule would have treated the exact &lt;code&gt;TypedHeader&lt;/code&gt; path as insufficient rather than inventing one. It would have treated the generic-parameter claim as contradicted rather than paraphrasing it as a removal. And it would have kept retrieval inefficiency visible instead of hiding it inside a large context.&lt;/p&gt;
&lt;h2 id=&quot;the-limited-conclusion&quot;&gt;The limited conclusion&lt;/h2&gt;
&lt;p&gt;The Axum A/B investigation does not establish a universal law about evidence volume, nor does it provide a general benchmark for Agent Loop. It establishes a practical engineering warning: retrieval coverage and claim correctness are separate dimensions. In this trace, the evidence-discipline session collected substantially more retrieval output while still requiring a separate review to catch the final-answer interpretation errors.&lt;/p&gt;
&lt;p&gt;An agent can search too little, search redundantly, or search effectively and still synthesize an unsupported claim. Reliability improves when the system records not just what was retrieved, but what each source actually licenses the agent to say.&lt;/p&gt;
&lt;p&gt;In other words, the goal is not maximum evidence. It is the smallest evidence bundle that makes every important claim auditable.&lt;/p&gt;
&lt;p&gt;Related: &lt;a href=&quot;/blog/evidence-to-claim-verification/&quot;&gt;Evidence-to-Claim Verification: Why LLM Reviewers Still Fail&lt;/a&gt; and &lt;a href=&quot;/blog/independent-agent-testing/&quot;&gt;Testing AI Agents With Independent Agents&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>The Crash Window: Recovering AI Agent Actions Without Blind Retries</title>
        <published>2026-11-27T00:00:00+00:00</published>
        <updated>2026-11-27T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/crash-window-execution-journal/"/>
        <id>https://iamavk.info/blog/crash-window-execution-journal/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/crash-window-execution-journal/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/crash-window-execution-journal/hero.png&quot; alt=&quot;Abstract illustration of an agent action crossing a crash boundary&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Tool-using agents are often described as loops: ask the model, execute a tool, append the result, and continue. The difficult part starts when the tool has a side effect and the process dies at exactly the wrong moment.&lt;/p&gt;
&lt;p&gt;Suppose a tool writes to an external database. The write commits. Before the agent can persist that the call completed, the process crashes. On restart, the journal still says &lt;code&gt;running&lt;/code&gt;. Should the runtime execute the tool again?&lt;/p&gt;
&lt;p&gt;No—at least not by default. The external effect may already exist.&lt;/p&gt;
&lt;p&gt;I turned that boundary into a regression test in Agent Loop. The test is committed as &lt;code&gt;395e8e0&lt;/code&gt; and uses two real subprocesses plus SQLite. It does not prove exactly-once execution. It proves a narrower and more useful invariant: after an unresolved call crosses a crash boundary, recovery records an unknown outcome and does not blindly redispatch it.&lt;/p&gt;
&lt;p&gt;The execution-recovery work was subsequently hardened in commit &lt;code&gt;cc577d2&lt;/code&gt;. That follow-up adds a second boundary: if compaction fails while a tool call is still unresolved, the runtime must not lose the execution record, mark the call complete, or dispatch it again after resume. It also adds session-integrity checks for journal/message consistency and durable replacement behavior.&lt;/p&gt;
&lt;p&gt;The recovery work was then completed with two additional regression commits. &lt;code&gt;b9f3325&lt;/code&gt; verifies that truncated or structurally partial session files fail closed instead of becoming resumable. &lt;code&gt;798a589&lt;/code&gt; adds real MCP stdio disconnect fixtures before and after an external side-effect commit; both cases are classified as &lt;code&gt;unknown_outcome&lt;/code&gt; and non-retryable, while the external marker is checked independently of the MCP response. These commits extend the evidence from a local subprocess crash to transport loss and partial-session recovery. They add regression coverage; they do not claim that arbitrary external systems provide exactly-once semantics.&lt;/p&gt;
&lt;p&gt;The current &lt;code&gt;v0.1.0&lt;/code&gt; reliability contract also makes the session-write boundary explicit: session data is validated, written to a same-directory temporary file, file-synced, atomically replaced, and followed by a directory fsync. A failure before replacement preserves the previous session; after replacement, a complete new session is observable. This is a bounded durability guarantee, not a claim about power loss on every filesystem, storage device, or hardware cache.&lt;/p&gt;
&lt;p&gt;The broader ExecutionJournal failure-boundary audit is now complete. Its final report records &lt;code&gt;1,312 passed&lt;/code&gt; and covers pending-persist failure, persistence before dispatch, ambiguous external actions, crash after side effect, terminal-save failure, failed compaction with unresolved calls, restart recovery without redispatch, atomic persistence and fsync, partial or corrupt sessions, MCP timeout, and MCP connection loss both before and after an external side effect. The resulting invariant is broader than the original crash test: unresolved execution state remains separate from the summary and is never retried automatically.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/crash-window-execution-journal/crash-window.svg&quot; alt=&quot;The crash window between an external commit and the durable journal write&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-state-machine&quot;&gt;The state machine&lt;/h2&gt;
&lt;p&gt;The runtime keeps a small &lt;code&gt;ExecutionJournal&lt;/code&gt; for each tool call. A record moves through explicit states:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;PENDING -&amp;gt; RUNNING -&amp;gt; COMPLETED&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                    -&amp;gt; FAILED&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                    -&amp;gt; UNKNOWN&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Terminal states carry an outcome string. A completed call has a result; a failed call has a failure description; an unknown call has a recovery explanation.&lt;/p&gt;
&lt;p&gt;The important state is &lt;code&gt;UNKNOWN&lt;/code&gt;. It is not a synonym for failure. It means the runtime cannot establish whether the external operation took effect.&lt;/p&gt;
&lt;p&gt;That distinction is easy to lose if recovery is designed around the assumption that the journal is authoritative. The journal is durable application state, but it is not a transaction spanning every external system the tool can touch.&lt;/p&gt;
&lt;h2 id=&quot;the-test-boundary&quot;&gt;The test boundary&lt;/h2&gt;
&lt;p&gt;Process A registers a tool call and moves it to &lt;code&gt;RUNNING&lt;/code&gt;. The dispatched action inserts one row into an external SQLite database and commits. Immediately after the commit—and before the journal can record a terminal result—the process exits with a non-zero status.&lt;/p&gt;
&lt;p&gt;The assertions after process A exits are deliberately concrete:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the session file contains the tool call in &lt;code&gt;running&lt;/code&gt; state;&lt;/li&gt;
&lt;li&gt;the external database contains exactly one effect;&lt;/li&gt;
&lt;li&gt;no terminal journal write has been assumed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Process B loads the session. Its provider returns a normal stopping response, while the test replaces dispatch with a function that fails if called. Recovery then materializes the unresolved call as &lt;code&gt;unknown&lt;/code&gt; and persists the explanation.&lt;/p&gt;
&lt;p&gt;The second process must satisfy three conditions:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;effect_count   = 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;dispatch_count = 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;status         = unknown&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The test also checks that the recovered tool message says the action was not re-executed. This makes the no-replay behavior visible in both the execution journal and the conversation history.&lt;/p&gt;
&lt;h2 id=&quot;independent-review-and-the-compaction-boundary&quot;&gt;Independent review and the compaction boundary&lt;/h2&gt;
&lt;p&gt;This recovery boundary was also raised during independent review. Tim Urista pointed out that unresolved tool calls must remain distinguishable from completed calls after older conversation messages disappear during compaction. His proposed invariant is deliberately narrower than “the agent remembers everything”: execution state should remain outside the prose summary, identified by the call ID and terminal outcome, so recovery can rebuild only the context needed for the next decision.&lt;/p&gt;
&lt;p&gt;That concern connects directly to this test, but it is not the test&#39;s evidence. The subprocess regression checks a separate failure boundary: an external effect commits before the terminal journal record is written. On recovery, the call remains visible as &lt;code&gt;UNKNOWN&lt;/code&gt;, the effect is not replayed, and the runtime does not invent a successful result. The same separation also means that compaction must not turn an unresolved action into either &lt;code&gt;COMPLETED&lt;/code&gt; or an automatic retry.&lt;/p&gt;
&lt;p&gt;Urista later asked for the execution-journal revision and suggested checking the external effect and journal together after recovery: no replay, no invented success, and the unresolved call still visible. The original external-effect test is &lt;code&gt;395e8e0&lt;/code&gt;; the subsequent recovery-hardening revision is &lt;code&gt;cc577d2&lt;/code&gt;. The comments are independent review and test-design input, not additional experimental results.&lt;/p&gt;
&lt;p&gt;References: &lt;a rel=&quot;external&quot; href=&quot;https://timothy-urista.medium.com/the-hard-truncation-fallback-in-compaction-is-a-fallback-not-unlimited-memory-suggests-a-useful-72abc420e02c&quot;&gt;compaction and unresolved tool calls&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://timothy-urista.medium.com/separating-conversation-compaction-from-execution-tracking-addresses-the-boundary-i-meant-046ce715aa17&quot;&gt;separating compaction from execution tracking&lt;/a&gt;, and &lt;a rel=&quot;external&quot; href=&quot;https://timothy-urista.medium.com/the-pinned-revision-makes-the-recovery-boundary-clear-recover-unresolved-preserves-unknown-rather-af06d8860c78&quot;&gt;the pinned revision and recovery test&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;the-updated-compaction-boundary&quot;&gt;The updated compaction boundary&lt;/h2&gt;
&lt;p&gt;The new regression forces the summary provider to fail while a resumed session still contains a &lt;code&gt;RUNNING&lt;/code&gt; tool call. The fallback path must preserve the execution record, recover it as &lt;code&gt;UNKNOWN&lt;/code&gt; with an &lt;code&gt;unknown_outcome&lt;/code&gt; category, append a tool message explaining that the action was not re-executed, and send no dispatch request.&lt;/p&gt;
&lt;p&gt;This is a different failure boundary from the SQLite crash test. The crash test asks what happens when an external effect may already have committed. The compaction test asks what happens when the narrative summary cannot be produced. Both cases require the same conservative invariant: execution state lives outside the prose summary, and uncertainty must remain visible after recovery.&lt;/p&gt;
&lt;p&gt;The current focused validation covers the execution-state, session-integrity, compaction, and MCP contract regression modules. The last directly reproduced subset passed 100 tests; the completed audit report records 1,312 passed across the full failure-boundary scope. These results describe the checked checkout and audit run; they are not a general reliability score for Agent Loop.&lt;/p&gt;
&lt;h2 id=&quot;why-retrying-is-unsafe&quot;&gt;Why retrying is unsafe&lt;/h2&gt;
&lt;p&gt;A retry policy can be correct for a pure function. It is not automatically correct for an external action. A process crash removes the runtime&#39;s knowledge of where it stopped; it does not roll back a database commit, an email, a payment, a deployment, or a file upload.&lt;/p&gt;
&lt;p&gt;The dangerous shortcut is:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;if status == RUNNING after restart:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    run the tool again&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That converts uncertainty into a second side effect. If the first attempt succeeded, the retry duplicates it. If the first attempt failed before doing anything, the retry may be reasonable—but the runtime cannot safely know which case occurred without tool-specific idempotency or an external reconciliation check.&lt;/p&gt;
&lt;p&gt;The conservative rule is therefore: recover the record, do not dispatch it, and require verification before relying on the result.&lt;/p&gt;
&lt;h2 id=&quot;what-the-test-does-not-guarantee&quot;&gt;What the test does not guarantee&lt;/h2&gt;
&lt;p&gt;This is not a distributed transaction protocol. It does not provide exactly-once execution. It does not make arbitrary tools idempotent. It does not prove that a tool can never partially apply an effect. It does not replace reconciliation with the external system.&lt;/p&gt;
&lt;p&gt;It proves one recovery behavior for one controlled failure boundary. A production tool still needs an idempotency key, a read-after-write verification path, or a domain-specific reconciliation procedure if the operation matters.&lt;/p&gt;
&lt;p&gt;The word “unknown” is doing real engineering work here. It prevents the runtime from manufacturing certainty that it does not possess.&lt;/p&gt;
&lt;h2 id=&quot;why-the-subprocess-matters&quot;&gt;Why the subprocess matters&lt;/h2&gt;
&lt;p&gt;An in-process test could simulate an exception, but it would not exercise session durability across a process boundary. The regression starts a fresh interpreter for recovery, reloads the serialized state, and verifies the behavior against a second process.&lt;/p&gt;
&lt;p&gt;The test also has explicit timeouts for both subprocesses. A recovery test that hangs can hide the very failure it is meant to catch. The focused validation passed 100 tests across execution-state recovery, session integrity, compaction, and MCP disconnect boundaries; the completed audit reported 1,312 passed across the full boundary matrix. &lt;code&gt;git diff --check&lt;/code&gt; passed in the updated checkout before the later local working-tree change.&lt;/p&gt;
&lt;p&gt;Those results are evidence about the tested code path—not a general reliability score for the whole agent runtime.&lt;/p&gt;
&lt;h2 id=&quot;the-design-lesson&quot;&gt;The design lesson&lt;/h2&gt;
&lt;p&gt;Durable state is valuable precisely because it makes uncertainty explicit. A good recovery path answers three separate questions:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;What did the runtime persist before the crash?&lt;/li&gt;
&lt;li&gt;What external effects can be observed now?&lt;/li&gt;
&lt;li&gt;What action is safe to take without duplicating an unknown effect?&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The journal can answer the first. A reconciliation tool or external system must answer the second. The safest default for the third is often “do nothing automatically.”&lt;/p&gt;
&lt;p&gt;That may sound less productive than retrying, but it is more truthful. An agent that pauses with an auditable &lt;code&gt;UNKNOWN&lt;/code&gt; record is recoverable. An agent that silently duplicates a side effect has destroyed information about what happened.&lt;/p&gt;
&lt;p&gt;Related: &lt;a href=&quot;/blog/a-successful-search-that-never-searched/&quot;&gt;A Successful Search That Never Searched&lt;/a&gt; and &lt;a href=&quot;/blog/independent-agent-testing/&quot;&gt;Testing AI Agents With Independent Agents&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>A Successful Search That Never Searched</title>
        <published>2026-11-20T00:00:00+00:00</published>
        <updated>2026-11-20T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/a-successful-search-that-never-searched/"/>
        <id>https://iamavk.info/blog/a-successful-search-that-never-searched/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/a-successful-search-that-never-searched/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/a-successful-search-that-never-searched/hero.png&quot; alt=&quot;Abstract illustration of a retrieval request resolving to known and unknown sources&quot; /&gt;&lt;/p&gt;
&lt;p&gt;An agent can be wrong because a model misunderstood evidence. It can also be wrong because the evidence request never reached a source at all.&lt;/p&gt;
&lt;p&gt;That second failure is deceptively quiet. A search tool receives an explicit source selector, resolves it to zero indexed sources, and returns a successful response containing zero hits. From the caller&#39;s point of view, the operation looks like a legitimate empty search. The agent may continue, summarize the absence of results, and make a decision with missing context.&lt;/p&gt;
&lt;p&gt;I found this boundary while investigating Context Mode 2.2.0. The important result was not a benchmark number or a claim about every retrieval system. It was a small, reproducible protocol observation: through the real MCP stdio JSON-RPC interface, an unknown explicit source selector produced &lt;code&gt;isError: false&lt;/code&gt; and a zero-hit message instead of a source-resolution error.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/a-successful-search-that-never-searched/unknown-source.svg&quot; alt=&quot;Decision flow for an explicit source selector&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-failure-mode&quot;&gt;The failure mode&lt;/h2&gt;
&lt;p&gt;The minimal shape of the request was:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;json&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;method&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;tools/call&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;params&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;    &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;ctx_search&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;    &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;arguments&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;source&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;https://example.invalid/missing&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;queries&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; [&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;alpha-token&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The observed response was equivalent to:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;isError: false&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;No hits in scope &amp;quot;worktree&amp;quot; across 0 source(s).&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That response conflates two different states:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The requested source exists and contains no matching evidence.&lt;/li&gt;
&lt;li&gt;The requested source does not resolve in the active search scope.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Those states have different meanings. The first is a valid negative result. The second is a failed precondition.&lt;/p&gt;
&lt;p&gt;The test was differential: the same request was sent through the actual MCP boundary and compared against the baseline behavior. That matters because a unit test around an internal resolver could miss serialization, adapter, or JSON-RPC behavior. The failure was observed where an agent observes it.&lt;/p&gt;
&lt;h2 id=&quot;why-zero-hits-is-not-harmless&quot;&gt;Why “zero hits” is not harmless&lt;/h2&gt;
&lt;p&gt;An empty result normally carries useful information. It can mean that a known document does not contain the requested term. But an empty result from zero sources says nothing about the document.&lt;/p&gt;
&lt;p&gt;This distinction is especially important in agent workflows because tool success often becomes an implicit control signal. A model is more likely to treat a successful tool reply as evidence that the requested operation happened. If the tool returns a normal success envelope for an invalid source, the model has to infer an infrastructure failure from a sentence that does not say one occurred.&lt;/p&gt;
&lt;p&gt;That is a poor contract. It makes “the source is empty,” “the query is too narrow,” “the source is out of scope,” and “the selector is misspelled” look similar.&lt;/p&gt;
&lt;h2 id=&quot;the-narrow-correction&quot;&gt;The narrow correction&lt;/h2&gt;
&lt;p&gt;The safest correction is small: validate an explicit source selector after the existing resolver has handled all supported aliases and special selectors. If the selector is not a known indexed source or a valid memory selector, return a structured &lt;code&gt;UNKNOWN_SOURCE&lt;/code&gt; error.&lt;/p&gt;
&lt;p&gt;The intended diagnostic is along these lines:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;UNKNOWN_SOURCE:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;source &amp;quot;...&amp;quot; is not indexed in scope &amp;quot;worktree&amp;quot;.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Use the exact label or source handle returned by ctx_index/ctx_search,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;or omit source to search available content.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The key compatibility rule is that a known source with no matching query must remain a normal successful zero-hit result. The change should not broaden a search, retry with another source, or reject valid forms such as source handles, &lt;code&gt;#N&lt;/code&gt; aliases, normalized path labels, memory selectors, or the reserved &lt;code&gt;decision&lt;/code&gt; selector.&lt;/p&gt;
&lt;h2 id=&quot;the-scope-caveat&quot;&gt;The scope caveat&lt;/h2&gt;
&lt;p&gt;There is a subtle distinction between “unknown globally” and “not available in this scope.” A source may exist in another worktree while being unavailable to a &lt;code&gt;worktree&lt;/code&gt; search in the current one. A global existence check can therefore still misclassify an out-of-scope source.&lt;/p&gt;
&lt;p&gt;The experiment establishes the unknown-selector behavior. It does not, by itself, settle the final product semantics for every cross-worktree case. That contract needs to be decided explicitly. One reasonable design is to keep &lt;code&gt;UNKNOWN_SOURCE&lt;/code&gt; as the public error while stating the effective scope in the diagnostic. Another is to introduce a separate &lt;code&gt;SOURCE_OUT_OF_SCOPE&lt;/code&gt; code. The important property is that neither case should silently become a successful search over zero sources.&lt;/p&gt;
&lt;p&gt;There is another practical constraint: the source selector is user-controlled input. Error formatting should keep the message bounded and single-line, rather than allowing arbitrary newlines or very long selectors to corrupt the diagnostic or journal output.&lt;/p&gt;
&lt;h2 id=&quot;what-this-test-does-and-does-not-prove&quot;&gt;What this test does—and does not—prove&lt;/h2&gt;
&lt;p&gt;It proves a concrete contract mismatch in the tested baseline: an explicit selector that resolved to no indexed sources was reported as a successful zero-hit search through the real MCP interface.&lt;/p&gt;
&lt;p&gt;It does not prove that every empty retrieval result is a bug. It does not prove that an agent will always hallucinate after this response. It does not justify automatically retrying with a broader query. The reproduction itself was run against the Context Mode 2.2.0 baseline; it is not evidence about behavior after a later fix.&lt;/p&gt;
&lt;p&gt;The reproduction is tracked in &lt;a rel=&quot;external&quot; href=&quot;https://github.com/mksglu/context-mode/issues/1390&quot;&gt;Context Mode issue #1390&lt;/a&gt;. As of the latest issue update, the maintainer marked it &lt;code&gt;fixed-on-main&lt;/code&gt; and linked fix commit &lt;code&gt;334094e&lt;/code&gt;; the issue says the change ships in the next release. That status should not be read as proof that every released version already contains the fix, so the tested baseline and the post-fix behavior remain separate claims.&lt;/p&gt;
&lt;p&gt;The broader lesson is simple: retrieval tools should fail closed at the boundary between source resolution and search. “No matches” is evidence. “No source” is an error.&lt;/p&gt;
&lt;p&gt;This article belongs to a series about making tool-using agents observable and conservative. The next article looks at the analogous failure on the action side: what to do when an external effect commits immediately before the agent process crashes.&lt;/p&gt;
&lt;p&gt;Related: &lt;a href=&quot;/blog/crash-window-execution-journal/&quot;&gt;The Crash Window: Recovering AI Agent Actions Without Blind Retries&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Building an AWStats-Style Log Analyzer in Rust</title>
        <published>2026-11-06T00:00:00+00:00</published>
        <updated>2026-11-06T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/"/>
        <id>https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/hero.png&quot; alt=&quot;Abstract illustration of a Rust log analyzer processing server data&quot; /&gt;&lt;/p&gt;
&lt;p&gt;At first glance, a web log analyzer sounds like a weekend project. Read a line, parse an IP address and a URL, increment a counter, and put the result into an HTML table.&lt;/p&gt;
&lt;p&gt;That description holds up until the first log rotation, the first slightly unusual Apache format, or the first attempt to update yesterday&#39;s report without counting yesterday&#39;s requests again. It gets more interesting when the analyzer needs to survive a misbehaving native plugin.&lt;/p&gt;
&lt;p&gt;Those are the problems I ended up working through while building &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats&quot;&gt;Stats&lt;/a&gt;, an experimental, AWStats-style web log analyzer in Rust. I recently published &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/releases/tag/v0.1.2&quot;&gt;v0.1.2&lt;/a&gt;. This is a look at the design choices and the work behind that release, including the parts that were less straightforward than the initial parser.&lt;/p&gt;
&lt;h2 id=&quot;why-revisit-awstats-style-log-analysis&quot;&gt;Why revisit AWStats-style log analysis?&lt;/h2&gt;
&lt;p&gt;In November 2025, the maintainers of &lt;a rel=&quot;external&quot; href=&quot;https://github.com/eldy/awstats&quot;&gt;AWStats&lt;/a&gt; announced that the project was deprecated and no longer actively developed. AWStats has a long history, and its author, Laurent Destailleur, deserves credit for maintaining it for so many years. The upstream notice recommends &lt;a rel=&quot;external&quot; href=&quot;https://matomo.org/log-analytics/&quot;&gt;Matomo Log Analytics&lt;/a&gt; as a supported migration destination.&lt;/p&gt;
&lt;p&gt;I didn&#39;t set out to argue against that recommendation. But the announcement brought a different question to mind: what would a smaller, Rust-based implementation of &lt;em&gt;selected&lt;/em&gt; AWStats workflows look like today?&lt;/p&gt;
&lt;p&gt;There are still situations where parsing access logs locally is useful. You may want reports without adding JavaScript to a site, or you may already have log files and a scheduled job that generates HTML. A local analyzer can also be operated without introducing an external analytics service. None of that makes it automatically more private or more accurate: the logs themselves, the derived identifiers, and how long they are retained still matter.&lt;/p&gt;
&lt;p&gt;The aim for Stats is deliberately narrower than “replace AWStats.” It is an independent implementation of some familiar configuration and reporting workflows, with explicit boundaries around what it can and cannot reproduce. It is not a fork, an official successor, or a drop-in replacement.&lt;/p&gt;
&lt;h2 id=&quot;compatibility-is-a-feature-with-a-boundary&quot;&gt;Compatibility is a feature with a boundary&lt;/h2&gt;
&lt;p&gt;Stats understands an AWStats-style &lt;code&gt;Key=Value&lt;/code&gt; configuration and supports several log-format families: Apache and Nginx custom formats, common access-log records, JSON-line input, Unix timestamps, and selected AWStats time formats. Optional time-zone normalization and MaxMind Country, City, and ASN databases are supported.&lt;/p&gt;
&lt;p&gt;That sounds like a broad list, but names are not the same as behavior. Accepting a directive does not mean reproducing all of AWStats&#39; semantics. Some presentation-only directives are accepted without changing output; unsupported directives are reported through validation diagnostics. Stats also does &lt;strong&gt;not&lt;/strong&gt; read AWStats&#39; original historical databases.&lt;/p&gt;
&lt;p&gt;I prefer documenting that distinction to silently promising compatibility. If two tools both show a “visits” column, it is tempting to assume that the underlying counting rules are identical. They may not be. Visit and bot classifications include approximations; log formats, time zones, and session heuristics all affect the numbers.&lt;/p&gt;
&lt;p&gt;The &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/blob/master/docs/compatibility.md&quot;&gt;compatibility matrix&lt;/a&gt; is therefore part of the product, not a footnote. It identifies supported behavior and known differences, so a prospective user can make an informed decision before migrating anything.&lt;/p&gt;
&lt;h2 id=&quot;the-pipeline-parse-once-report-in-several-ways&quot;&gt;The pipeline: parse once, report in several ways&lt;/h2&gt;
&lt;p&gt;I split the implementation into a Cargo workspace rather than putting ingestion, storage, and presentation into one binary. The workspace has twelve packages. &lt;code&gt;stats-core&lt;/code&gt; handles parsing, normalized events, enrichment, storage, and report queries. &lt;code&gt;stats-cli&lt;/code&gt; coordinates command-line operations. There are also HTTP and CGI adapters, a terminal UI, and separate renderer plugins.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/architecture.svg&quot; alt=&quot;Stats processing path: frontends select input, stats-core normalizes events, SQLite stores aggregates, and a shared query API serves report consumers.&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The shared query layer keeps presentation code away from the details of SQLite storage.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The path is reasonably simple:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Read configuration and select an input log.&lt;/li&gt;
&lt;li&gt;Parse records and normalize event fields.&lt;/li&gt;
&lt;li&gt;Apply configured time-zone, GeoIP, and event-enrichment behavior.&lt;/li&gt;
&lt;li&gt;Update SQLite aggregates and ingestion checkpoints.&lt;/li&gt;
&lt;li&gt;Read report data through the shared query API.&lt;/li&gt;
&lt;li&gt;Produce static HTML, terminal output, or responses through the available frontends and configured renderers.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The shared query API is one of the more useful separations. The terminal summary, TUI, and HTML report should not each invent their own interpretation of stored aggregates. The presentation layer can change without rewriting ingestion or the storage schema.&lt;/p&gt;
&lt;p&gt;HTML is a static, multi-page report by default; an HTTP or CGI frontend can also read existing data. Optional external renderers produce HTML, JSON, or CSV when configured. These are distinct execution paths, not features that every binary automatically bundles.&lt;/p&gt;
&lt;h2 id=&quot;log-rotation-is-where-simple-counters-stop-being-simple&quot;&gt;Log rotation is where simple counters stop being simple&lt;/h2&gt;
&lt;p&gt;The natural first implementation of an analyzer is to read a file from the beginning. That is fine for a one-off report. It is not fine for a scheduled job on a busy website.&lt;/p&gt;
&lt;p&gt;Stats&#39; normal &lt;code&gt;update&lt;/code&gt; command is cumulative. It processes unseen input and retains its existing aggregates. To decide whether a previously processed range can be trusted, the updater uses checkpoints tied to the source file&#39;s identity and content. A replacement, truncation, or changed processed prefix must not be treated as an ordinary append.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/incremental-processing.svg&quot; alt=&quot;Incremental ingestion checks file identity and processed content before appending, and uses replay when an existing checkpoint is no longer safe.&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;An ordinary update follows the active log. Historical imports are an explicit operation.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;There are two cases worth keeping separate:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The daily update.&lt;/strong&gt; Read the configured active log, use the checkpoint where valid, and add new events. A normal update does not go hunting through all of the rotated archives in the directory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Historical backfill.&lt;/strong&gt; When I explicitly want to import existing history, &lt;code&gt;backfill&lt;/code&gt; discovers conventional numbered rotations such as &lt;code&gt;access.log.1&lt;/code&gt; and compressed archives like &lt;code&gt;.2.gz&lt;/code&gt;, &lt;code&gt;.3.bz2&lt;/code&gt;, or &lt;code&gt;.4.xz&lt;/code&gt;. It processes them oldest to newest. Individual files can also be supplied explicitly.&lt;/p&gt;
&lt;p&gt;Keeping those operations distinct matters. Running backfill on every timer tick may repeatedly open old archives, while using &lt;code&gt;rebuild&lt;/code&gt; during a multi-file import would intentionally clear derived aggregates. Neither is the same thing as an incremental update.&lt;/p&gt;
&lt;p&gt;There is also a practical boundary to the guarantee: checkpointing protects against specific file changes and repeated unchanged input. It is not a claim of a distributed, exactly-once event-processing system. An administrator still needs a sensible rotation schedule, readable archives, and a backup strategy.&lt;/p&gt;
&lt;h2 id=&quot;why-sqlite-and-what-i-gave-up&quot;&gt;Why SQLite, and what I gave up&lt;/h2&gt;
&lt;p&gt;The default storage layout uses one SQLite database for aggregate data and processing state. That choice suits a local analyzer: the state is on disk, inspectable, and does not require another server just to run a scheduled analysis job.&lt;/p&gt;
&lt;p&gt;The implementation uses write-ahead logging (WAL), a busy timeout, and explicit maintenance commands. SQLite still has concurrency and operational constraints; choosing it does not make every update or migration free of cost.&lt;/p&gt;
&lt;p&gt;For longer histories, Stats supports monthly or yearly partitions. The control database remains under &lt;code&gt;DirData/stats.sqlite3&lt;/code&gt;; event aggregates live in period-specific databases under &lt;code&gt;DirData/periods/&lt;/code&gt;. Queries can combine the period data for reports, while shared control and session state remains in one place.&lt;/p&gt;
&lt;p&gt;The trade-off is that cross-period reporting becomes more involved, and late or out-of-order events make some session-derived metrics approximate. I would rather surface that than present a “unique visits” number as if it were directly observed.&lt;/p&gt;
&lt;p&gt;There are explicit &lt;code&gt;backup&lt;/code&gt;, &lt;code&gt;optimize&lt;/code&gt;, &lt;code&gt;storage&lt;/code&gt;, &lt;code&gt;retention&lt;/code&gt;, and &lt;code&gt;migrate&lt;/code&gt; operations. Retention lists matching partitions before removing anything, can archive them, and requires an additional removal option to delete original partitions. Migration from the old single-database layout creates a backup before clearing and rebuilding derived aggregates. Those are operational decisions, not just database implementation details.&lt;/p&gt;
&lt;p&gt;One less obvious detail is privacy. Stats does not store raw client IP addresses as visit identifiers, but the deterministic FNV-1a aggregation keys are &lt;strong&gt;pseudonymous&lt;/strong&gt;, not cryptographic anonymization. They still deserve protection as analytics data. A log analyzer is not automatically privacy-safe just because it runs on your own machine.&lt;/p&gt;
&lt;h2 id=&quot;native-plugins-need-two-different-conversations&quot;&gt;Native plugins need two different conversations&lt;/h2&gt;
&lt;p&gt;Extensibility was another area where the easy implementation and the safe-enough implementation diverged.&lt;/p&gt;
&lt;p&gt;Stats supports native event enrichment and external report rendering, but the execution models differ significantly.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/building-awstats-style-log-analyzer-rust/plugin-isolation.svg&quot; alt=&quot;Plugin execution model: event enrichment uses trusted in-process callbacks, while report renderers run in supervised worker processes with bounded IPC.&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The worker boundary helps contain crashes. It is not a sandbox for untrusted code.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;An &lt;strong&gt;event-enrichment plugin&lt;/strong&gt; executes in the analyzer&#39;s process. It works with normalized fields such as the client address, method, URL, status, and byte count. Since it is native code running in-process, a bug in that callback can affect the analyzer. It must be treated as trusted deployment code.&lt;/p&gt;
&lt;p&gt;An external &lt;strong&gt;report-renderer plugin&lt;/strong&gt; goes through a short-lived supervised worker process. Requests and responses are limited to 8 MiB. The supervisor handles failures such as non-zero exits, aborts, segmentation faults, malformed or oversized responses, and timeouts. A failed rendering request can return a controlled error instead of taking down the CLI, HTTP, or CGI frontend.&lt;/p&gt;
&lt;p&gt;That is useful fault containment, but there are limits. The worker retains its operating-system privileges. It can access files allowed to that account and may spawn other processes; independently spawned descendants are not automatically cleaned up. There is no claim of OS-level sandboxing or safe execution of arbitrary third-party binaries.&lt;/p&gt;
&lt;p&gt;I think this distinction is worth spelling out because “runs in another process” can sound stronger than it is. It solves some failure modes. It does not solve the entire trust problem.&lt;/p&gt;
&lt;h2 id=&quot;getting-from-tests-to-a-release&quot;&gt;Getting from tests to a release&lt;/h2&gt;
&lt;p&gt;The project did not go straight from “compiles locally” to a clean public release. I had to spend time debugging timeouts and other issues during testing and CI. I am intentionally not assigning causes to individual earlier failures here without their corresponding logs and fixes in front of me; a final green workflow does not explain every failed run that came before it.&lt;/p&gt;
&lt;p&gt;What the final runs &lt;em&gt;do&lt;/em&gt; establish is more concrete.&lt;/p&gt;
&lt;p&gt;For the &lt;code&gt;v0.1.2&lt;/code&gt; release commit (&lt;code&gt;d09f0bb&lt;/code&gt;), the GitHub Actions checks completed formatting, workspace checking, &lt;strong&gt;123 passing tests&lt;/strong&gt;, and Clippy with warnings treated as errors. The CI run also completed a dependency audit and produced a Linux release bundle. The separate release workflow repeated the core Rust checks and tests, built the optimized binary, packaged it, and verified the archive checksum.&lt;/p&gt;
&lt;p&gt;The worker tests are particularly relevant to the architecture discussion. They cover direct worker failure scenarios including abort, SIGSEGV, invalid replies, oversized messages, and timeouts; additional tests check that CLI, HTTP, and CGI requests fail in a controlled way and that subsequent requests can succeed. That is much stronger evidence for the worker boundary than a diagram alone.&lt;/p&gt;
&lt;p&gt;The published workflow also confirmed that &lt;code&gt;softprops/action-gh-release&lt;/code&gt; uploaded both &lt;code&gt;stats-0.1.2-linux-x86_64.tar.gz&lt;/code&gt; and the corresponding &lt;code&gt;.sha256&lt;/code&gt; file to the existing &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/releases/tag/v0.1.2&quot;&gt;GitHub Release&lt;/a&gt;. The checksum verification passed before upload. This is a real release artifact, not just an Actions artifact hidden in a CI run.&lt;/p&gt;
&lt;p&gt;There are still things that result does &lt;strong&gt;not&lt;/strong&gt; prove. The supplied release log does not show an independent clean-machine installation and end-to-end smoke test of the downloaded archive. Nor does passing the worker fault tests make arbitrary native plugins safe. I prefer to keep the evidence at the level of what was actually exercised.&lt;/p&gt;
&lt;p&gt;The Linux bundle includes the analyzer binary and deployment materials. Optional dynamic HTML, JSON, and CSV renderer libraries are &lt;strong&gt;not&lt;/strong&gt; included and must be built or installed separately when needed. That detail belongs in the deployment documentation, because “the archive built successfully” and “every optional plugin is shipped” are two different statements.&lt;/p&gt;
&lt;p&gt;The &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/actions/runs/42228582518&quot;&gt;CI run&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/releases/tag/v0.1.2&quot;&gt;release page&lt;/a&gt;, and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/blob/master/docs/deployment.md&quot;&gt;deployment instructions&lt;/a&gt; make the verification trail available to anyone who wants to inspect it.&lt;/p&gt;
&lt;h2 id=&quot;what-works-today-and-what-i-am-not-claiming&quot;&gt;What works today, and what I am not claiming&lt;/h2&gt;
&lt;p&gt;At &lt;code&gt;v0.1.2&lt;/code&gt;, Stats has implemented parsing and validation for a documented subset of log formats and AWStats-style configuration, incremental SQLite aggregation, optional period partitioning, several reporting frontends, optional GeoIP data, and a native renderer plugin path with tested worker failure handling.&lt;/p&gt;
&lt;p&gt;It is still experimental. There is no promise of full AWStats directive coverage, identical historical statistics, drop-in database migration, a stable public plugin API, or production suitability for every deployment. This is also not a speed comparison against AWStats or other analytics systems; I have not presented a reproducible benchmark that would justify one.&lt;/p&gt;
&lt;p&gt;A useful next step is to exercise the packaged binary in a clean installation, validate reports against representative real-world log samples, and continue making the compatibility boundaries more explicit. Those are more valuable than adding another unchecked claim to the README.&lt;/p&gt;
&lt;h2 id=&quot;what-i-took-away&quot;&gt;What I took away&lt;/h2&gt;
&lt;p&gt;The parser was the obvious part of the project. The less obvious work was deciding what to do when the input changes underneath you, keeping historical data understandable, and making plugin failures predictable enough that a report request does not take down its frontend.&lt;/p&gt;
&lt;p&gt;The release process added another lesson: tests and packaging are not a formality at the end. Debugging them is part of implementing the system. A green workflow is useful evidence, but it is a snapshot of the checks that actually ran, not a certificate that nothing else can fail.&lt;/p&gt;
&lt;p&gt;For me, Stats is most interesting as a collection of those boundaries: compatibility without pretending to be identical, incremental processing without promising magic, and process isolation without calling it a security sandbox.&lt;/p&gt;
&lt;p&gt;The source is available under &lt;strong&gt;GPL-3.0-or-later&lt;/strong&gt; at &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats&quot;&gt;avkghost/stats&lt;/a&gt;. If you want to try the example log, the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/blob/master/docs/getting-started.md&quot;&gt;getting-started guide&lt;/a&gt; documents the commands and the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/stats/blob/master/docs/compatibility.md&quot;&gt;compatibility matrix&lt;/a&gt; explains what to expect.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>From Ready Pods to a Working System: Building a Distributed Kafka Chat Architecture</title>
        <published>2026-11-04T00:00:00+00:00</published>
        <updated>2026-11-04T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/"/>
        <id>https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/hero.png&quot; alt=&quot;Abstract illustration of a distributed Kafka chat architecture&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;An experimental architecture case study, November 2026. The system described here is a laboratory proof of concept, not a production deployment or a compliance certification.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Source code and reproducible lab:&lt;/strong&gt; &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture&quot;&gt;Kafka Chat — Experimental Distributed Messaging Architecture on GitHub&lt;/a&gt;. The repository contains the Rust gateway, Helm/deployment manifests, bootstrap scripts, architecture notes, and verification matrix. See also the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/docs/architecture.md&quot;&gt;architecture documentation&lt;/a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/docs/publication-verification-matrix.md&quot;&gt;verification matrix&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;A green Kubernetes dashboard is reassuring. It is not a proof that a system works.&lt;/p&gt;
&lt;p&gt;In this project, I saw an identity provider report Ready and serve valid OpenID Connect discovery metadata, yet remain effectively unmanageable because its initial administrator credential had not been made available. I also saw a healthy PostgreSQL high-availability cluster reject a properly configured client: the server certificate identified &lt;code&gt;spilo.example.org&lt;/code&gt;, not the Kubernetes Service DNS name the application used. Later, an OAuth project and API application existed, but machine-to-machine authentication still failed with &lt;code&gt;invalid_client&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;None of those incidents was a Kubernetes readiness failure. All were application-level integration failures.&lt;/p&gt;
&lt;p&gt;I wanted to explore a more interesting question than “Can I get Kafka and PostgreSQL running on Kubernetes?”: &lt;strong&gt;Can I bootstrap an isolated, security-conscious tenant stack from scratch, rerun that bootstrap without breaking existing state, and demonstrate authenticated messaging and permission revocation across independent application instances?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The resulting PoC combines Rust/Axum WebSocket gateways, Kafka in KRaft mode, PostgreSQL managed by Patroni and the Zalando Operator, encrypted Longhorn volumes, RustFS, ZITADEL, and a PostgreSQL-backed room access-control model. A final authenticated test delivered messages in both directions between separate gateway Pods and showed that an authenticated principal removed from a room could neither publish nor receive subsequent messages.&lt;/p&gt;
&lt;p&gt;This article covers the architecture, the evidence, the costs, and the failures that changed the design. It deliberately separates &lt;strong&gt;what was observed&lt;/strong&gt; from &lt;strong&gt;what still needs testing&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id=&quot;contents&quot;&gt;Contents&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#1-goals-assumptions-and-boundaries&quot;&gt;1. Goals, assumptions, and boundaries&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#2-architecture-a-dedicated-stack-inside-one-namespace&quot;&gt;2. Architecture: a dedicated stack inside one namespace&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#3-a-reproducible-bootstrap-not-a-migration-exercise&quot;&gt;3. A reproducible bootstrap, not a migration exercise&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#4-kafka-ha-kraft-replication-and-cross-instance-fan-out&quot;&gt;4. Kafka HA: KRaft, replication, and cross-instance fan-out&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#5-postgresql-ha-patroni-and-one-primary-for-authorization&quot;&gt;5. PostgreSQL HA: Patroni and one primary for authorization&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#6-encrypted-longhorn-volumes-and-the-boundary-of-backup-evidence&quot;&gt;6. Encrypted Longhorn volumes and the boundary of backup evidence&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#7-replica-only-read-routing-and-its-consistency-trade-off&quot;&gt;7. Replica-only read routing and its consistency trade-off&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#8-zitadel-when-ready-did-not-mean-usable&quot;&gt;8. ZITADEL: when Ready did not mean usable&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#9-rust-jwt-validation-and-postgresql-room-authorization&quot;&gt;9. Rust JWT validation and PostgreSQL room authorization&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#10-what-the-end-to-end-tests-actually-proved&quot;&gt;10. What the end-to-end tests actually proved&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#11-failure-modes-that-improved-the-design&quot;&gt;11. Failure modes that improved the design&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/#12-trade-offs-limits-and-what-i-would-do-differently&quot;&gt;12. Trade-offs, limits, and what I would do differently&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;1-goals-assumptions-and-boundaries&quot;&gt;1. Goals, assumptions, and boundaries&lt;/h2&gt;
&lt;p&gt;The starting point was a small Rust/Kafka chat prototype. Its initial terminal-oriented design was useful for learning about producers, consumers, topics, and asynchronous work, but it was not enough to reason about identity, room membership, multi-instance fan-out, recovery, or operational isolation. The experiment therefore evolved into a WebSocket backend running on Kubernetes.&lt;/p&gt;
&lt;p&gt;The target was &lt;strong&gt;one namespace per tenant&lt;/strong&gt;, with a dedicated application stack inside it. For this lab, the namespace is &lt;code&gt;kafka-chat&lt;/code&gt;. A tenant gets its own Kafka HA deployment, PostgreSQL HA cluster, identity provider, Rust gateways, storage credentials, and namespace-scoped configuration. Operators that manage multiple installations remain cluster-wide or operator-scoped shared components.&lt;/p&gt;
&lt;p&gt;This choice prioritizes isolation and an understandable operational boundary over hardware efficiency. It is not the only valid multi-tenant design. A shared Kafka cluster plus tenant-specific topics, or a shared PostgreSQL cluster with separate databases and strict access controls, might be more economical. I wanted to test the stronger dedicated-stack model first.&lt;/p&gt;
&lt;p&gt;The success criteria were concrete: a clean bootstrap that brings up the whole tenant infrastructure; a second bootstrap run that preserves credentials and persistent identities; valid internal TLS identities; a real machine-issued OAuth access token; independent JWT verification; two authenticated gateway replicas; cross-instance messaging; and a room membership revocation test on already connected clients.&lt;/p&gt;
&lt;p&gt;Several goals were intentionally out of scope. The PoC does not demonstrate availability across independent physical hosts, a throughput or latency service-level objective, complete disaster recovery of the live database, browser-based Authorization Code with PKCE, or compliance with a particular government or industry framework. Three replicas on one physical machine do not create three independent physical failure domains.&lt;/p&gt;
&lt;h2 id=&quot;2-architecture-a-dedicated-stack-inside-one-namespace&quot;&gt;2. Architecture: a dedicated stack inside one namespace&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/architecture.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/architecture.svg&quot; alt=&quot;Kafka Chat tenant architecture: dedicated services in the kafka-chat namespace and shared cluster operators&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The gateway is the public-facing application boundary in the design, although this experiment used internal connectivity rather than a production public ingress. Clients authenticate with ZITADEL-issued JWT access tokens. The Rust/Axum process validates those tokens using OIDC discovery and JWKS, then asks PostgreSQL whether the verified identity may enter a room, publish, or receive a message.&lt;/p&gt;
&lt;p&gt;Kafka carries room events between gateways. Each gateway needs to receive events relevant to its locally connected clients. This matters: if gateway replicas simply share one Kafka consumer group, Kafka will divide partitions among them; it will not automatically broadcast every room event to every replica. The implementation uses separate gateway consumer groups for instance-level fan-out, with the acknowledged cost that every gateway may consume events it ultimately does not deliver locally.&lt;/p&gt;
&lt;p&gt;PostgreSQL has a different job. It is the authority for room membership and permissions, not the transport for chat events in this PoC. Its primary handles security-sensitive ACL decisions. The read-only HAProxy route is a separate experiment for non-authoritative reads; it is explicitly not placed in the authorization path.&lt;/p&gt;
&lt;p&gt;The namespace arrangement also makes naming predictable. A service such as &lt;code&gt;kafka-chat-postgres&lt;/code&gt; has a DNS identity derived from its namespace and cluster domain. Kafka advertised broker names, PostgreSQL certificate SANs, Secrets, Helm values, and OIDC endpoints must all refer to the same actual deployment topology. A namespace is more than an organizational label once clients validate server identities.&lt;/p&gt;
&lt;p&gt;There is a critical caveat: &lt;strong&gt;a Kubernetes namespace is not, by itself, a complete security boundary&lt;/strong&gt;. A defensible tenant model also needs ServiceAccounts and RBAC, network restrictions, admission and Pod Security policies, resource controls, Secret-management rules, and separation of privileged operators. This lab exercises part of that pattern; it does not prove complete hostile-tenant isolation.&lt;/p&gt;
&lt;h2 id=&quot;3-a-reproducible-bootstrap-not-a-migration-exercise&quot;&gt;3. A reproducible bootstrap, not a migration exercise&lt;/h2&gt;
&lt;p&gt;An early iteration placed services in several namespaces, including &lt;code&gt;kafka-ha&lt;/code&gt; and a PostgreSQL Service under &lt;code&gt;database&lt;/code&gt;. That made the setup harder to reason about and directly contributed to mismatched DNS assumptions. I initially considered preserving and migrating every experimental database, Kafka offset, and PVC to the final namespace. For a disposable lab, that was the wrong optimization.&lt;/p&gt;
&lt;p&gt;The real objective was &lt;strong&gt;bootstrap&lt;/strong&gt;, not migration. The final approach discarded verified disposable lab state and rebuilt the dedicated stack under &lt;code&gt;kafka-chat&lt;/code&gt;. The reset was narrowly scoped: identify which resources belong to this experiment before deleting anything, and leave shared operators and unrelated workloads alone.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/bootstrap-flow.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/bootstrap-flow.svg&quot; alt=&quot;Bootstrap dependency graph from shared prerequisites through verified OAuth, authenticated gateways, E2E checks, and repeat bootstrap&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: Dependency gates rather than just Pod readiness. The repeat run tests reuse of existing persistent identities; the arrows show logical prerequisites, not a single serialized command sequence.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The repository acquired two entrypoints:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;scripts/reset-kafka-chat-lab.sh&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;scripts/bootstrap-kafka-chat-lab.sh&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The reset command is a separate, guarded operation. Ordinary bootstrap execution does not imply data destruction. The bootstrap checks its prerequisites, creates or reuses tenant resources, provisions credentials and certificates, deploys stateful components in dependency order, and validates resulting connectivity. It is not a general-purpose cloud provisioning framework; it is a reproducible workflow for this lab.&lt;/p&gt;
&lt;p&gt;The most important verification was running it &lt;strong&gt;twice&lt;/strong&gt;. The first real clean run produced a healthy three-member PostgreSQL cluster, a three-broker Kafka HA cluster, a Ready RustFS tenant, and an initialized ZITADEL. The second run reused existing credentials, PostgreSQL PVCs, and Kafka cluster identity rather than generating a new cluster or invalidating Secrets. In a later iteration, provisioning also created the OAuth machine identities needed to start authenticated gateways.&lt;/p&gt;
&lt;p&gt;Idempotency has limits: observing a successful repeat run does not prove that every imaginable partial failure can be recovered automatically. It does show that the normal rerun does not casually destroy the identity of already-running stateful services. That is substantially stronger evidence than a successful &lt;code&gt;helm template&lt;/code&gt; or a manifest dry-run.&lt;/p&gt;
&lt;p&gt;A good bootstrap script is also a contract between components. The order matters: cert-manager must be able to issue the service identity before a strict TLS client connects; the PostgreSQL primary and required roles must exist before ZITADEL initializes; the ZITADEL administrator must be usable before automation can create machine accounts; and the machine token must be issued and checked before an auth-required gateway can start. Treating each stage as an explicit dependency produces actionable failures instead of a long sequence of apparently healthy but unusable Pods.&lt;/p&gt;
&lt;p&gt;The contract should be testable without publishing Secrets. An operational report can safely include namespace, release names, object readiness, certificate SANs, token claim names, audience-match results, and Kafka topic replication settings. It should not include actual PATs, machine secrets, JWTs, or Longhorn encryption material. That distinction is particularly important if the experiment will be described in a public article.&lt;/p&gt;
&lt;p&gt;There is another reproducibility lesson here. If the automation requires a person to create a TLS Secret or copy an OAuth audience into an environment variable before each install, the documented procedure is incomplete. Those steps must either be automated or made explicit, validated external dependencies. The final flow derives its verified audience from a real token rather than inventing an identifier.&lt;/p&gt;
&lt;h2 id=&quot;4-kafka-ha-kraft-replication-and-cross-instance-fan-out&quot;&gt;4. Kafka HA: KRaft, replication, and cross-instance fan-out&lt;/h2&gt;
&lt;p&gt;The Kafka portion uses three KRaft combined broker/controller instances. The final lab creates fresh KRaft metadata on a clean install. The chat topic was verified with replication factor &lt;strong&gt;3&lt;/strong&gt; and minimum in-sync replicas &lt;strong&gt;2&lt;/strong&gt;. Kafka communication used TLS and SASL authentication, and the deployment checked broker discovery as well as the bootstrap Service. The latter is easy to miss: after the initial metadata request, a Kafka client contacts advertised broker addresses, not only the hostname provided at startup.&lt;/p&gt;
&lt;p&gt;For durability semantics, &lt;code&gt;min.insync.replicas=2&lt;/code&gt; has its intended effect when paired with an appropriate producer acknowledgement configuration such as &lt;code&gt;acks=all&lt;/code&gt;. The topic setting alone is not a guarantee that every client uses that acknowledgement policy. Kafka also provides ordering within partitions, not an unconditional total order across an application.&lt;/p&gt;
&lt;p&gt;The room-event model is important to the chat design. Keying events by room can keep that room&#39;s records together under a stable partitioning scheme, while message IDs and bounded deduplication help manage duplicate observations at the gateway. That still does not establish exactly-once delivery to WebSocket clients. A Kafka produce acknowledgement means the broker accepted the record under its configured durability conditions; it does not prove that every connected browser or CLI client received it.&lt;/p&gt;
&lt;p&gt;For cross-gateway delivery, each Rust instance consumes the event stream and distributes eligible room events to its own connected subscribers. The design avoids the common error of expecting a shared consumer group to behave as a broadcast mechanism. It does incur extra consumption and authorization work as the number of gateways grows. At larger scale, room-aware routing, partition ownership, or a different fan-out topology might become attractive; this PoC did not benchmark those alternatives.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/kafka-fanout.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/kafka-fanout.svg&quot; alt=&quot;Kafka chat fan-out: two gateways publish to the same topic but consume through separate gateway-specific consumer groups&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: Separate consumer groups deliver each event to both gateway processes; each gateway then filters for locally connected and authorized subscribers. This is not a client delivery-acknowledgement protocol.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/cross-gateway-delivery-sequence.svg&quot; alt=&quot;Cross-gateway message delivery sequence: broker acknowledgment, gateway authorization, and WebSocket send&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Earlier migration experiments also explored message and offset transfer between separate Kafka clusters, including a rollback path. That was useful operational research, but it is not part of the final clean-bootstrap contract. Fresh test data and consumer groups are appropriate for a laboratory reset. The article&#39;s core result is the verified new cluster and authenticated event flow, not a zero-loss production migration guarantee.&lt;/p&gt;
&lt;p&gt;Kafka HA here means &lt;strong&gt;replicated Kafka processes and topic data&lt;/strong&gt;. It does not imply survival of a physical host outage. The earlier lab evidence placed several replicas on the same worker; the final experiments did not demonstrate independent physical fault domains. Kafka&#39;s own documentation also discourages combined broker/controller roles for critical production environments. Both caveats matter when interpreting a three-node topology running on one machine.&lt;/p&gt;
&lt;h2 id=&quot;5-postgresql-ha-patroni-and-one-primary-for-authorization&quot;&gt;5. PostgreSQL HA: Patroni and one primary for authorization&lt;/h2&gt;
&lt;p&gt;PostgreSQL 16 is managed by the Zalando PostgreSQL Operator with Patroni. The clean bootstrap verified one leader and two streaming replicas. The observed WAL replay lag in that check was &lt;strong&gt;zero&lt;/strong&gt;, and the three instance PVCs were encrypted. A point-in-time observation of zero lag says the replicas were caught up during the check; it does not make asynchronous replication synchronous or prove that every replica read is current at every instant.&lt;/p&gt;
&lt;p&gt;The application needs two distinct databases or logical database contexts: ZITADEL&#39;s own state and Kafka Chat&#39;s room model. Room membership uses a deliberately small schema: &lt;code&gt;rooms&lt;/code&gt; and &lt;code&gt;room_members&lt;/code&gt;, with an authenticated identity represented by the validated issuer and subject, plus a room role such as owner or member. The backend uses parameterized queries and transactional membership changes.&lt;/p&gt;
&lt;p&gt;A separate read-only routing experiment showed why “send reads to a replica” is not a sufficient policy. The authorization path cannot tolerate a stale membership decision after someone has been removed from a room. Even a short replication delay might allow a revoked account to be treated as an active member. This is why &lt;strong&gt;ACL reads go to the primary&lt;/strong&gt;, despite the availability and scaling trade-off.&lt;/p&gt;
&lt;p&gt;Strict PostgreSQL TLS became an unexpectedly important part of this work. The running Spilo instance initially presented a certificate associated with &lt;code&gt;spilo.example.org&lt;/code&gt;, while the application connected through a Kubernetes Service in another namespace. The Rust client correctly rejected the identity mismatch. Disabling hostname verification would have made the integration test pass for the wrong reason.&lt;/p&gt;
&lt;p&gt;The fix was to produce a certificate with SANs matching the actual Service DNS names, configure Zalando&#39;s custom TLS Secret, and set the required &lt;code&gt;spiloFSGroup&lt;/code&gt;. The clean namespace-local bootstrap then verified the primary Service connection using full hostname and CA verification. That fix addressed the infrastructure naming model rather than simply suppressing a client error.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/tls-service-identity.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/tls-service-identity.svg&quot; alt=&quot;TLS DNS failure and correction: mismatched Spilo certificate identity rejected, correct PostgreSQL Service SAN accepted&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: The failure was expected behavior from a strict TLS client. The right fix was a certificate matching the real internal Service identity, not disabling verification.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;6-encrypted-longhorn-volumes-and-the-boundary-of-backup-evidence&quot;&gt;6. Encrypted Longhorn volumes and the boundary of backup evidence&lt;/h2&gt;
&lt;p&gt;The storage experiment used a Longhorn StorageClass with encryption enabled. The per-PVC key controller creates distinct, PVC-bound encryption Secrets. Earlier checks inspected the node&#39;s device mappings and confirmed that a filesystem was mounted over a &lt;code&gt;dm-crypt&lt;/code&gt;/LUKS device, rather than merely trusting a Kubernetes annotation. Separate PVC keys were compared and found distinct. The final PostgreSQL HA deployment reported three encrypted PVCs.&lt;/p&gt;
&lt;p&gt;This is useful evidence for storage-at-rest encryption, but it must be interpreted precisely. It does not prove resistance to a fully compromised Kubernetes administrator, unrestricted access to encryption Secrets, or a compromised node that already has a volume mounted. Key lifetime, recovery, rotation, and off-host key protection would need their own threat model.&lt;/p&gt;
&lt;p&gt;The stack also has an isolated RustFS tenant, managed by an existing RustFS Operator. The S3-compatible endpoint and dedicated bucket were tested in earlier experiments as a target for WAL-G. The most revealing recovery exercise did not start as a backup success. A base-backup attempt initially failed because the restored fixture lacked the necessary checkpoint WAL segment. Another attempt ran into an AppArmor-related signal permission mismatch: a &lt;code&gt;kubectl exec&lt;/code&gt; process and the PostgreSQL service process did not share the same confinement context. A corrected disposable fixture ran PostgreSQL and WAL-G in the same appropriate process tree.&lt;/p&gt;
&lt;p&gt;The resulting isolated WAL-G experiment passed checkpoint creation, WAL archiving, backup/fetch, recovery startup, and SQL integrity checks. It restored known test records to a consistent point. &lt;strong&gt;That does not mean the running PostgreSQL HA cluster completed a disaster-recovery drill.&lt;/strong&gt; In particular, recovery of the active cluster, promotion after loss of its primary, and replay of additional post-backup WAL beyond the demonstrated restore point were not proven.&lt;/p&gt;
&lt;p&gt;There are two distinct claims: encrypted persistent volumes were observed, and a disposable backup/restore fixture worked. A mature DR story would add scheduled backup validation, retention checks, an independent restore environment, tested key recovery, and measurable recovery point and recovery time objectives.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/backup-evidence.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/backup-evidence.svg&quot; alt=&quot;WAL-G evidence boundary: an isolated fixture was backed up to RustFS, restored and SQL-checked; full live HA disaster recovery was not tested&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: A consistent isolated restore is meaningful evidence, but it must not be relabeled as verified live HA disaster recovery.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;7-replica-only-read-routing-and-its-consistency-trade-off&quot;&gt;7. Replica-only read routing and its consistency trade-off&lt;/h2&gt;
&lt;p&gt;The HAProxy proof of concept explored read scaling without silently falling back to the PostgreSQL leader. Its backend discovery used Kubernetes DNS and Patroni&#39;s replica health endpoint with a strict zero-lag query, represented in the fixture by &lt;code&gt;/replica?lag=0&amp;amp;replication_state=streaming&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;It is easy to overstate what a health check means. A replica can become stale after it was checked and before the next query. A lag threshold of zero at sampling time is not a linearizability guarantee. The goal of this router was narrower: choose nodes that currently advertise the required replica state and reject connections when no eligible replica exists.&lt;/p&gt;
&lt;p&gt;A controlled backend fixture exercised actual client connections during health transitions. A failing backend was excluded after two unsuccessful checks, in approximately &lt;strong&gt;2–5 seconds&lt;/strong&gt;, and re-entered after two successful checks, in approximately &lt;strong&gt;4–8 seconds&lt;/strong&gt;. When all simulated replicas were unhealthy, the router failed closed. Earlier SQL tests reached both replica IPs and confirmed &lt;code&gt;pg_is_in_recovery()=true&lt;/code&gt;, not the primary.&lt;/p&gt;
&lt;p&gt;These are useful but bounded observations. The delays describe a controlled fixture, not a statistically significant availability benchmark. Actual WAL lag injection, production-style Patroni failover, and EndpointSlice replacement were not established as end-to-end results. The HAProxy container also needed a higher memory limit after an earlier 64 MiB limit produced exit code 137; the tested configuration used 128 MiB.&lt;/p&gt;
&lt;p&gt;This component illustrates a broader trade-off. Replica routing is attractive for eventually consistent reads and reports. It is the wrong default for a security check that must act on freshly committed membership changes. The application therefore keeps its authorization workload on the primary even though an RO router exists.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/postgres-routing.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/postgres-routing.svg&quot; alt=&quot;PostgreSQL reads separated by consistency: room ACL queries always reach the primary; HAProxy sends eligible ordinary reads only to streaming replicas&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: The RO route fails closed when no replica is eligible. The health check is sampled; it cannot promise that a replica remains current after the check.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;8-zitadel-when-ready-did-not-mean-usable&quot;&gt;8. ZITADEL: when Ready did not mean usable&lt;/h2&gt;
&lt;p&gt;For identity, I chose to test &lt;strong&gt;ZITADEL as an alternative to Keycloak&lt;/strong&gt;, not to run a head-to-head benchmark. Keycloak was not deployed into this Kafka Chat lab. There is consequently no measured ZITADEL-versus-Keycloak comparison in the results.&lt;/p&gt;
&lt;p&gt;The deployed ZITADEL application version was &lt;strong&gt;4.19.2&lt;/strong&gt;, using Helm chart &lt;strong&gt;10.4.0&lt;/strong&gt;, and a dedicated database in the existing PostgreSQL HA cluster. The instance used an internal issuer, originally verified as &lt;code&gt;http://zitadel.kafka-chat.svc.cluster.local:8080&lt;/code&gt;. That HTTP issuer was an intentional internal laboratory constraint, not a recommendation for a publicly exposed production IdP.&lt;/p&gt;
&lt;p&gt;The first identity-provider experiment reached a deceptive state. Helm setup completed, the application Pod was Ready, and discovery and JWKS returned HTTP 200. Database inspection showed an instance, an &lt;code&gt;iam-admin&lt;/code&gt; machine user, and a human administrator. Yet the expected bootstrap PAT or usable administrator credential had not been materialized. The instance could advertise its keys while the automation could not create an OAuth client or test authentication.&lt;/p&gt;
&lt;p&gt;The important issue was the &lt;strong&gt;first-instance credential lifecycle&lt;/strong&gt;: rerunning Helm against an already initialized ZITADEL database did not recreate missing initial access. For a disposable lab, rather than writing a complex recovery procedure, I reset only ZITADEL&#39;s experiment-specific state and configured first-instance bootstrap correctly. The clean run created the administrative credentials, supported an OAuth client-credentials test, and passed independent JWT signature, issuer, audience, and expiration checks.&lt;/p&gt;
&lt;p&gt;The next failure was different. The full single-namespace bootstrap automatically created a ZITADEL project and API applications, but token issuance returned &lt;code&gt;invalid_client&lt;/code&gt;. The cause was established: the provisioning step tried to use &lt;strong&gt;API-application credentials for a machine-user &lt;code&gt;client_credentials&lt;/code&gt; flow&lt;/strong&gt;. The corrected implementation used ZITADEL User Service v2 machine users with their own persisted secrets. Real tokens were issued, and JWT verification passed. The audience was read and validated from actual token content and passed through Helm as a &lt;strong&gt;string&lt;/strong&gt; rather than allowing its long numeric identifier to be reinterpreted.&lt;/p&gt;
&lt;p&gt;This distinction is worth more than a simple “ZITADEL worked” status. &lt;strong&gt;An OAuth resource server, an API application, a service-account identity, and an administrator PAT are different objects with different responsibilities.&lt;/strong&gt; Treating them as interchangeable can create an installation that appears configured but cannot authenticate the client one intends to use.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/oidc-token-path.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/oidc-token-path.svg&quot; alt=&quot;Machine OAuth path: initialize ZITADEL, provision machine identity, issue real JWT, validate JWKS and claims in gateway, then check room membership on PostgreSQL primary&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Figure: Authentication ends at verified token identity; room authorization is a separate, primary-backed decision. Human browser login with PKCE was not part of this test.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In a prior test, idle ZITADEL usage was observed at approximately 7 milliCPU and 104 MiB of memory; earlier observations had different values. These are snapshots from particular runs, not a load benchmark or a claim about predictable tenant operating cost. The database, storage, and auxiliary processes also consume resources.&lt;/p&gt;
&lt;h2 id=&quot;9-rust-jwt-validation-and-postgresql-room-authorization&quot;&gt;9. Rust JWT validation and PostgreSQL room authorization&lt;/h2&gt;
&lt;p&gt;The Rust/Axum gateway does not depend on a vendor-specific ZITADEL SDK. Instead, it uses OIDC discovery and JWKS to verify signed access tokens. The implemented validation path caches discovery metadata and keys, restricts algorithms to RS256, checks issuer and audience, validates expiration and &lt;code&gt;nbf&lt;/code&gt;, requires a valid subject and key identifier, and refreshes keys when rotation or an unknown &lt;code&gt;kid&lt;/code&gt; requires it.&lt;/p&gt;
&lt;p&gt;Authentication only answers &lt;em&gt;who is making this request?&lt;/em&gt; It does not answer &lt;em&gt;is this subject allowed to read or publish in this room?&lt;/em&gt; Room authorization is therefore stored separately in PostgreSQL. The stable identity key is issuer plus subject, not a display name or a user ID supplied in the WebSocket payload.&lt;/p&gt;
&lt;p&gt;The gateway has distinct enforcement points for subscriptions, publishing, membership management, and outbound delivery. That last point was an intentional design decision. If a gateway checks a membership once when a client subscribes, removing that member from PostgreSQL would not necessarily stop messages flowing over an already open socket. The implementation performs revocation-aware authorization before outbound delivery.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/revocation.svg&quot;&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/revocation.svg&quot; alt=&quot;Cross-gateway authorization and revocation: PostgreSQL primary is authoritative at publish and delivery checks&quot; /&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/kafka-chat-distributed-messaging-architecture/revocation-sequence.svg&quot; alt=&quot;Authorization revocation sequence: membership commit, denied publish, and denied delivery&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Two kinds of consistency meet in this path. Kafka provides event transport and its own delivery/replay semantics; PostgreSQL supplies the current authorization state. A Kafka event can be present in the consumer stream while the recipient is no longer authorized to receive it. The gateway must enforce that distinction at the moment it decides to send the event to its WebSocket client.&lt;/p&gt;
&lt;p&gt;This costs database work. Doing an ACL check at delivery time can increase read traffic, latency, and sensitivity to database availability. In-memory caching could improve performance but introduce a revocation window unless carefully invalidated. For a correctness-oriented PoC, primary-backed checks gave a clear model to test before optimizing.&lt;/p&gt;
&lt;p&gt;The WebSocket Bearer-token path was tested with machine clients. Browser clients do not universally permit arbitrary Authorization headers in native WebSocket handshakes. A production browser experience would need a deliberately designed login and socket-establishment flow, potentially using a short-lived WebSocket ticket. That work, along with interactive Authorization Code plus PKCE, was outside this experiment.&lt;/p&gt;
&lt;h2 id=&quot;10-what-the-end-to-end-tests-actually-proved&quot;&gt;10. What the end-to-end tests actually proved&lt;/h2&gt;
&lt;p&gt;The end-to-end test used &lt;strong&gt;real ZITADEL-issued JWT access tokens&lt;/strong&gt; for two distinct machine subjects and connected them to &lt;strong&gt;different gateway Pods&lt;/strong&gt;. The test did not rely on the Kubernetes Service randomly choosing separate replicas. The gateway instances interacted through Kafka HA, while permission decisions were read from the PostgreSQL primary.&lt;/p&gt;
&lt;p&gt;The first authenticated rehearsal established successful delivery in both directions. An unauthorized subject was denied room access. The clean bootstrap could then be rerun without rebuilding the Kafka cluster, resetting PostgreSQL PVCs, or rotating machine secrets.&lt;/p&gt;
&lt;p&gt;The final targeted test kept both WebSocket clients connected while the room owner removed the other authenticated principal from the membership table. The revoked principal&#39;s new publish attempt was rejected; a new message published by the owner was not delivered to the revoked principal. A reconnect and resubscription attempt also failed. When membership was restored, room access worked again.&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Experiment&lt;/th&gt;&lt;th&gt;Observation&lt;/th&gt;&lt;th&gt;What it does &lt;strong&gt;not&lt;/strong&gt; prove&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Clean bootstrap&lt;/td&gt;&lt;td&gt;PostgreSQL HA, Kafka HA, RustFS and ZITADEL came up successfully&lt;/td&gt;&lt;td&gt;Robust recovery from every interrupted bootstrap step&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Second bootstrap&lt;/td&gt;&lt;td&gt;Credentials, PVCs and Kafka cluster identity reused&lt;/td&gt;&lt;td&gt;Exhaustive long-term idempotency under all failure modes&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Kafka HA&lt;/td&gt;&lt;td&gt;Three brokers; topic RF=3/min ISR=2&lt;/td&gt;&lt;td&gt;Independent physical-host availability&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;PostgreSQL HA&lt;/td&gt;&lt;td&gt;One leader, two streaming replicas; observed lag 0&lt;/td&gt;&lt;td&gt;Synchronous consistency for every replica read&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;PostgreSQL TLS&lt;/td&gt;&lt;td&gt;Correct Service SAN and trusted-CA verification&lt;/td&gt;&lt;td&gt;Complete PKI lifecycle or production certificate rotation tests&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OAuth / JWT&lt;/td&gt;&lt;td&gt;Real machine tokens; signature, issuer and audience validated&lt;/td&gt;&lt;td&gt;Human browser login or PKCE&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Cross-gateway messaging&lt;/td&gt;&lt;td&gt;Authenticated A→B and B→A delivery passed&lt;/td&gt;&lt;td&gt;Global exactly-once client delivery or throughput targets&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Revoked publish and delivery&lt;/td&gt;&lt;td&gt;Both denied after membership removal&lt;/td&gt;&lt;td&gt;Retroactive cancellation of already authorized, in-flight events&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Reconnect after revocation&lt;/td&gt;&lt;td&gt;Subscription denied; restoration later passed&lt;/td&gt;&lt;td&gt;Absence of all possible revocation races&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;WAL-G fixture&lt;/td&gt;&lt;td&gt;Backup/fetch and consistent restore passed&lt;/td&gt;&lt;td&gt;Full recovery drill of the active PostgreSQL HA cluster&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;The reported membership-change operation on the PostgreSQL primary took approximately &lt;strong&gt;5 ms locally&lt;/strong&gt; in the final revocation rehearsal. That is &lt;strong&gt;not&lt;/strong&gt; an end-to-end revocation latency measurement. No post-commit race was observed within the bounded test, but “none observed” is not a linearizability proof, and messages already in flight cannot be recalled retroactively. Failure injection for unavailable PostgreSQL ACL queries was not performed in that final rehearsal.&lt;/p&gt;
&lt;p&gt;A useful way to state the result is: &lt;strong&gt;removing an authenticated principal from a room must prevent tested subsequent publishing and outbound delivery, including across separate gateway instances, once the membership removal has committed.&lt;/strong&gt; That statement is narrower, and much stronger, than claiming “instant revocation” without a defined consistency model.&lt;/p&gt;
&lt;h2 id=&quot;11-failure-modes-that-improved-the-design&quot;&gt;11. Failure modes that improved the design&lt;/h2&gt;
&lt;p&gt;Several problems shaped the final architecture more than the initial successful deployments did.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Namespace drift broke identity assumptions.&lt;/strong&gt; Services spread across &lt;code&gt;database&lt;/code&gt;, &lt;code&gt;kafka-ha&lt;/code&gt;, and &lt;code&gt;kafka-chat&lt;/code&gt; made internal DNS, TLS SANs, client endpoint configuration, and ownership less predictable. The fix was a single namespace-local bootstrap and explicit exceptions for shared operators. No stateful cross-namespace migration was needed for a disposable experiment.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A successful Helm installation hid failed administrative initialization.&lt;/strong&gt; ZITADEL looked healthy from the outside but lacked usable bootstrap credentials. The improvement was to treat admin access and OAuth token issuance as functional readiness checks. A Pod becoming Ready is only the beginning.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A configured OAuth application was not a machine identity.&lt;/strong&gt; Creating projects and API applications did not make the machine &lt;code&gt;client_credentials&lt;/code&gt; request valid. The actual fix required the right ZITADEL user type and its persisted credentials, followed by an issued and independently verified token. The distinction belongs in deployment automation, not just in documentation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Correct TLS verification exposed a naming mistake.&lt;/strong&gt; PostgreSQL was healthy and reachable, yet the presented certificate did not match the Service DNS. The safe response was to fix certificate issuance and Spilo configuration, not to set a weaker client TLS mode. Strict verification is useful precisely because it makes such mistakes visible.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Backup success required recovery evidence.&lt;/strong&gt; WAL-G needed the necessary checkpoint WAL and a process arrangement compatible with AppArmor restrictions. A command returning success is not sufficient; an isolated restore with SQL integrity checks supplied stronger proof. Even that proof does not extend automatically to live-cluster DR.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Replica health is a sampled observation.&lt;/strong&gt; HAProxy excluded failing backends and failed closed in a controlled test, but periodic checks do not establish a consistency barrier. This reinforced the decision to keep security-sensitive ACL reads on the PostgreSQL primary.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Authorization must survive the WebSocket connection lifetime.&lt;/strong&gt; Authentication at handshake and ACL at subscription were not enough for revocation semantics. The final test specifically verified outbound filtering after a membership change, across two independently connected gateways.&lt;/p&gt;
&lt;p&gt;These failures are not isolated incidents. Together, they illustrate a broader engineering principle: &lt;strong&gt;deployment readiness must be demonstrated through verified application behavior, not infrastructure health alone.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;12-trade-offs-limits-and-what-i-would-do-differently&quot;&gt;12. Trade-offs, limits, and what I would do differently&lt;/h2&gt;
&lt;p&gt;The most obvious benefit of this design is its clear boundary. One tenant namespace contains the tenant&#39;s runtime components, Secrets, certificates, storage, and application-specific persistence. The bootstrap can create the stack from scratch, and a rerun can verify that it does not rotate foundational identities. Dedicated infrastructure also reduces the ways one tenant&#39;s application-level data and credentials can accidentally be mixed with another&#39;s.&lt;/p&gt;
&lt;p&gt;That separation is expensive. Three Kafka processes, three PostgreSQL instances, a separate IdP, application gateways, storage volumes, monitoring, and backup infrastructure are substantial overhead for a small tenant. A namespace per tenant does not eliminate shared cluster failure modes or the operational trust placed in cluster-wide controllers. It also multiplies upgrade, certificate, backup, and observability work across tenants. The correct economic answer may be a hybrid: dedicate stateful infrastructure to higher-isolation customers and use carefully designed shared clusters elsewhere.&lt;/p&gt;
&lt;p&gt;This also changes the questions to ask before assigning a tenant its own dedicated stack. What is the minimum sustainable footprint when Kafka, PostgreSQL, and an IdP are all allocated separately? How will application and operator upgrades be rolled out across hundreds of namespaces? Which components can share a trusted control plane without accidentally sharing tenant data? What is the plan for tenants with wildly different traffic profiles? The lab answers none of these quantitatively, but it makes the resource and operational questions impossible to ignore.&lt;/p&gt;
&lt;p&gt;The Kafka fan-out model has a similar scale trade-off. Separate consumer groups per gateway are straightforward and testable, but they amplify Kafka reads as gateway count rises. PostgreSQL checks on outbound delivery preserve revocation semantics more directly, but they add authorization queries and make the primary an important dependency. Neither approach has been load-tested here, so I cannot attach credible messages-per-second, p95 latency, or tenants-per-node claims.&lt;/p&gt;
&lt;p&gt;The lab also had an important physical limit. Logical replication, several Pods, and separate PVCs do not establish host-level HA when the underlying Kubernetes workers, storage replicas, or power/network components share a failure domain. A production interpretation would require fault-domain placement, validated restore and failover procedures, monitoring, capacity management, key recovery, and a more deliberate KRaft controller/broker topology.&lt;/p&gt;
&lt;p&gt;For identity, the tested flow was machine-to-machine OAuth. A real product would add browser Authorization Code with PKCE, deliberate WebSocket session establishment, user/session lifecycle rules, abuse prevention, and token revocation or introspection policy where required. Token signature verification and room membership are necessary components of application security, not a complete security assessment.&lt;/p&gt;
&lt;p&gt;If starting again, I would keep the same narrow success criteria but enforce them earlier: define the namespace contract first; generate certificates from actual Service identities; make initial IdP administration and real token issuance mandatory deployment checks; reserve primary reads for security decisions; and document the difference between an infrastructure health probe, a functional test, and a recovery drill.&lt;/p&gt;
&lt;p&gt;The resulting proof of concept did achieve its central goal: &lt;strong&gt;a fresh single-namespace stack, a safe bootstrap rerun, real OAuth machine identity, authenticated cross-instance chat delivery, and demonstrated room membership revocation&lt;/strong&gt;. The untested areas are not hidden behind green status badges. They are the next design questions, if this laboratory architecture ever needs to support real users and real availability targets.&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;verification-trail-and-reference-material&quot;&gt;Verification trail and reference material&lt;/h3&gt;
&lt;p&gt;The observations above are drawn from recorded lab runs and source-control milestones, rather than a third-party benchmark. Relevant public repository commits include:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/6cc9bb770f13fc06d75764916bdde9bc6b9de1d&quot;&gt;6cc9bb7&lt;/a&gt; (WAL-G recovery proof),&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/a81cc1b6431705f975bc113daa613c75ee377717&quot;&gt;a81cc1b&lt;/a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/e27214dd871f88bb176a0ac11b11dae7edf496b1&quot;&gt;e27214d&lt;/a&gt; (ZITADEL PoC and bootstrap correction),&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/e19cf69ea5c1b1bc85c4794fb7e8322245a286d2&quot;&gt;e19cf69&lt;/a&gt; (Rust JWT and room ACL),&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/9db96e9de8e4875de61d9720c56cc465699ac094&quot;&gt;9db96e9&lt;/a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/0447c9518e4a6e86314f2ed4fe794860b83afc77&quot;&gt;0447c95&lt;/a&gt; (single-namespace bootstrap),&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/72a068bac12e4a3a94581f8404d46d6481c152d0&quot;&gt;72a068b&lt;/a&gt; (machine OAuth and authenticated cross-gateway delivery),&lt;/li&gt;
&lt;li&gt;and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/commit/c862064223401e68281a08953166f7c612fdeb1b&quot;&gt;c862064&lt;/a&gt; (cross-gateway revocation).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In particular, the final authorization evidence is in &lt;code&gt;kafka-chat-room-revocation-20261103T193000Z.md&lt;/code&gt;, and the machine OAuth E2E evidence is in &lt;code&gt;kafka-chat-oidc-machine-e2e-20261103T192000Z.md&lt;/code&gt;. These report paths are local experiment artifacts retained locally; they are not included in the public repository or represented here as public download links.&lt;/p&gt;
&lt;p&gt;The complete &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture&quot;&gt;Kafka Chat source repository&lt;/a&gt; includes the published architecture and implementation. The public &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/README.md&quot;&gt;README&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/docs/architecture.md&quot;&gt;architecture documentation&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/docs/publication-verification-matrix.md&quot;&gt;verification matrix&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/src/auth.rs&quot;&gt;JWT implementation&lt;/a&gt;, and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/kafka-chat-architecture/blob/master/LICENSE&quot;&gt;Apache-2.0 license&lt;/a&gt; provide stable repository references. For specification and implementation context, consult the &lt;a rel=&quot;external&quot; href=&quot;https://kafka.apache.org/documentation/&quot;&gt;Apache Kafka KRaft documentation&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://opensource.zalando.com/postgres-operator/&quot;&gt;Zalando PostgreSQL Operator documentation&lt;/a&gt;, &lt;a rel=&quot;external&quot; href=&quot;https://longhorn.io/docs/&quot;&gt;Longhorn encryption documentation&lt;/a&gt;, and &lt;a rel=&quot;external&quot; href=&quot;https://zitadel.com/docs/guides/integrate/zitadel-apis/access-zitadel-apis&quot;&gt;ZITADEL API access documentation&lt;/a&gt;. Repository configuration and lab reports remain the authoritative source for exact commands and installed version details.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Building a Cloud-Agnostic Kubernetes Platform for Multi-Tenant Applications</title>
        <published>2026-10-27T00:00:00+00:00</published>
        <updated>2026-10-27T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/"/>
        <id>https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/hero.png&quot; alt=&quot;Abstract illustration of a cloud-agnostic multi-tenant Kubernetes platform&quot; /&gt;&lt;/p&gt;
&lt;p&gt;I wanted a repeatable Kubernetes environment where I could work on the harder parts of a multi-tenant platform: networking, storage isolation, tenant provisioning, and recovery.&lt;/p&gt;
&lt;p&gt;I also wanted to avoid designing everything around one cloud provider from day one. My starting point was a local KVM/libvirt lab. The longer-term plan is to provision infrastructure through provider APIs and move the machine configuration and application deployment into Ansible, Terraform, and Helm.&lt;/p&gt;
&lt;p&gt;This post covers &lt;strong&gt;Cluster Bootstrap v0.1.0&lt;/strong&gt;, the first tested baseline. It is a working infrastructure foundation, &lt;strong&gt;not yet a production-ready multi-tenant platform&lt;/strong&gt;.&lt;/p&gt;
&lt;h2 id=&quot;what-the-first-version-actually-builds&quot;&gt;What the first version actually builds&lt;/h2&gt;
&lt;p&gt;The bootstrap script runs on a Linux KVM host and provisions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A separate management VM running Rancher in Docker.&lt;/li&gt;
&lt;li&gt;Two RKE2 server VMs and three RKE2 worker VMs.&lt;/li&gt;
&lt;li&gt;A libvirt NAT network (&lt;code&gt;192.168.100.0/24&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Cilium as the CNI, with WireGuard encryption and kube-proxy replacement.&lt;/li&gt;
&lt;li&gt;Cilium Envoy Ingress on NodePorts &lt;code&gt;30080&lt;/code&gt; (HTTP) and &lt;code&gt;30443&lt;/code&gt; (HTTPS).&lt;/li&gt;
&lt;li&gt;Longhorn with an encrypted-by-default StorageClass.&lt;/li&gt;
&lt;li&gt;A small controller that generates a distinct encryption Secret for each encrypted PVC.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/architecture.svg&quot; alt=&quot;Cluster Bootstrap v0.1.0 architecture&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The diagram separates the Rancher management VM from the downstream RKE2 nodes. The network and storage boxes represent services running &lt;strong&gt;inside&lt;/strong&gt; the downstream cluster, not additional VMs.&lt;/p&gt;
&lt;p&gt;Rancher is intentionally outside the downstream RKE2 cluster. It runs in its own VM, while the five other VMs form the managed cluster.&lt;/p&gt;
&lt;p&gt;This is a &lt;strong&gt;lab deployment&lt;/strong&gt;. The baseline deliberately uses some conveniences, including self-signed TLS and relaxed SSH host-key checking, that should not be carried into production.&lt;/p&gt;
&lt;h2 id=&quot;why-start-with-bash-and-kvm&quot;&gt;Why start with Bash and KVM?&lt;/h2&gt;
&lt;p&gt;The first goal was to prove the sequence of operations before introducing more abstraction.&lt;/p&gt;
&lt;p&gt;The script creates or reuses VM disks, sets up cloud-init, starts the management and RKE2 machines, waits for services, imports the cluster into Rancher, and configures storage. It also handles host-side details that are easy to overlook when focusing only on Kubernetes YAML: libvirt network definitions, VM image permissions, orphaned disks, and the order in which services become available.&lt;/p&gt;
&lt;p&gt;I chose a simple Bash bootstrap first because I wanted to see the full process and its failure modes. Terraform and Ansible are planned, but neither is part of this baseline.&lt;/p&gt;
&lt;h3 id=&quot;bootstrap-order-and-dependencies&quot;&gt;Bootstrap order and dependencies&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/bootstrap-sequence.svg&quot; alt=&quot;Bootstrap sequence and dependencies&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The sequence is based on the order in the frozen script: management VM and Rancher first, then downstream RKE2 VMs; Cilium configuration is written &lt;strong&gt;before&lt;/strong&gt; the first RKE2 server starts. After all five nodes become Ready, the script imports the cluster into Rancher and configures Longhorn and the PVC-key controller. The diagram shows the major dependencies, not every retry, wait, or API request.&lt;/p&gt;
&lt;h3 id=&quot;a-networking-problem-before-kubernetes-even-started&quot;&gt;A networking problem before Kubernetes even started&lt;/h3&gt;
&lt;p&gt;One early issue was the libvirt network itself. The script now handles migration from an older routed network (&lt;code&gt;cluster-routed&lt;/code&gt;) to the NAT network (&lt;code&gt;cluster-nat&lt;/code&gt;). It checks whether the network belongs to the lab, stops affected lab VMs before changing it, and refuses to remove a network used by an unrelated VM.&lt;/p&gt;
&lt;p&gt;The script also checks access to QCOW2 images by the &lt;code&gt;libvirt-qemu&lt;/code&gt; account. A VM disk can exist and still be unusable if QEMU cannot read the file or its backing image. These checks are small, but they matter for repeatable bootstrap.&lt;/p&gt;
&lt;p&gt;I would rather have the script stop with a clear error than silently replace a disk or alter an unrelated VM.&lt;/p&gt;
&lt;h2 id=&quot;rke2-cilium-and-encrypted-node-to-node-traffic&quot;&gt;RKE2, Cilium, and encrypted node-to-node traffic&lt;/h2&gt;
&lt;p&gt;For the downstream cluster I used RKE2 with Cilium instead of the default Canal configuration. The script disables the bundled ingress controller and kube-proxy, then supplies Cilium settings through RKE2&#39;s &lt;code&gt;HelmChartConfig&lt;/code&gt; &lt;strong&gt;before the first server starts&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The relevant configuration is compact:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;yaml&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;piVersion&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;elm.cattle.io/v1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;k&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;ind&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; H&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;elmChartConfig&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;etadata&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  n&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;ame&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; r&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;ke2-cilium&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  n&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;amespace&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; k&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;ube-system&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;s&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;pec&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  v&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;aluesContent&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; |&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;-&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    kubeProxyReplacement: true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    k8sServiceHost: localhost&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    k8sServicePort: 6443&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    encryption:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      enabled: true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      type: wireguard&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    ingressController:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      enabled: true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      default: true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      loadbalancerMode: shared&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      service:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;        type: NodePort&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;        insecureNodePort: 30080&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;        secureNodePort: 30443&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A detail worth keeping: on an existing RKE2 installation, the script &lt;strong&gt;does not attempt to switch the CNI in place&lt;/strong&gt;. It leaves the existing CNI and Helm configuration alone. Changing the CNI of a live cluster is a different migration problem.&lt;/p&gt;
&lt;p&gt;In the tested deployment, all five nodes reached &lt;code&gt;Ready&lt;/code&gt;. Cilium ran on every node, with two operator replicas. A Cilium encryption status check reported WireGuard, the &lt;code&gt;cilium_wg0&lt;/code&gt; interface, and four peers from one agent.&lt;/p&gt;
&lt;p&gt;That confirms WireGuard was enabled and peers were present. It is &lt;strong&gt;not&lt;/strong&gt; a claim that every possible inter-node or host-network traffic path was independently captured and verified.&lt;/p&gt;
&lt;h3 id=&quot;ingress-and-network-policy&quot;&gt;Ingress and network policy&lt;/h3&gt;
&lt;p&gt;The Cilium Envoy Ingress configuration exposed HTTP and HTTPS through fixed NodePorts. The observed HTTP test returned &lt;code&gt;200 OK&lt;/code&gt;, with &lt;code&gt;server: envoy&lt;/code&gt;, on port &lt;code&gt;30080&lt;/code&gt; across all five node IPs.&lt;/p&gt;
&lt;p&gt;A separate NetworkPolicy test allowed HTTP before an ingress-deny policy and timed out after it was applied. That is a useful functional check of policy enforcement, although it is not a comprehensive tenant-isolation audit.&lt;/p&gt;
&lt;h2 id=&quot;encrypted-persistent-storage-with-longhorn&quot;&gt;Encrypted persistent storage with Longhorn&lt;/h2&gt;
&lt;p&gt;The next problem was storage. Multi-tenant applications need persistent volumes, and I wanted encryption enabled by default rather than relying on each application to request it.&lt;/p&gt;
&lt;p&gt;The script installs Longhorn chart &lt;code&gt;1.13.0&lt;/code&gt; using the RKE2 Helm controller. On the cluster nodes it prepares &lt;code&gt;open-iscsi&lt;/code&gt;, &lt;code&gt;cryptsetup&lt;/code&gt;, and other required packages. It then creates a default StorageClass named &lt;code&gt;longhorn-encrypted&lt;/code&gt;, with three replicas and encryption enabled.&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;yaml&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;piVersion&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; s&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;torage.k8s.io/v1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;k&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;ind&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; S&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;torageClass&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;etadata&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  n&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;ame&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; l&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;onghorn-encrypted&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  a&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;nnotations&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;    s&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;torageclass.kubernetes.io/is-default-class&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;p&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;rovisioner&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; d&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;river.longhorn.io&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;llowVolumeExpansion&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; true&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;p&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;arameters&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  n&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;umberOfReplicas&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  e&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;ncrypted&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-stage-secret-name&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.name}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-stage-secret-namespace&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.namespace}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-publish-secret-name&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.name}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-publish-secret-namespace&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.namespace}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-expand-secret-name&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.name}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;  c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#22863A, #85E89D);&quot;&gt;si.storage.k8s.io/node-expand-secret-namespace&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; $&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{pvc.namespace}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&quot;generating-a-key-for-each-pvc&quot;&gt;Generating a key for each PVC&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/pvc-encryption-lifecycle.svg&quot; alt=&quot;Per-PVC encryption key lifecycle&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The key reconciler runs in the &lt;code&gt;storage-security&lt;/code&gt; namespace and polls for PVCs using the encrypted StorageClass. It creates Secrets in the &lt;strong&gt;PVC namespace&lt;/strong&gt;, not in its own namespace. This is a provisioning-time key mechanism; it does not implement off-cluster key backup or automatic key rotation.&lt;/p&gt;
&lt;p&gt;A small Python controller watches for PVCs using &lt;code&gt;longhorn-encrypted&lt;/code&gt;. For each one, it creates a Secret with the same name &lt;strong&gt;in the PVC&#39;s namespace&lt;/strong&gt;. The key is generated using Python&#39;s &lt;code&gt;secrets.token_urlsafe(48)&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The controller also records the PVC UID in the Secret annotation &lt;code&gt;storage.lab/pvc-uid&lt;/code&gt;. If the Secret already exists, it does not overwrite it. If the stored PVC UID differs from the current PVC UID, it logs a blocked mismatch rather than reusing the Secret.&lt;/p&gt;
&lt;p&gt;That behavior is important. Replacing a volume&#39;s encryption key during an ordinary reconciliation would make its data inaccessible. Reusing a key after a PVC was deleted and recreated under the same name would also be an unsafe assumption.&lt;/p&gt;
&lt;p&gt;The controller is intentionally simple: it polls PVCs every three seconds, has a single replica, and uses cluster-scoped permissions to list PVCs and read/create Secrets. Its list request is capped at 500 PVCs; if the API indicates pagination, the controller fails the scan rather than processing an incomplete list. It is a lab component, &lt;strong&gt;not a finished production key-management service&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id=&quot;verifying-the-encryption-not-just-the-yaml&quot;&gt;Verifying the encryption, not just the YAML&lt;/h3&gt;
&lt;p&gt;The test went beyond checking the StorageClass flag:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A 1 GiB PVC became &lt;code&gt;Bound&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Its encryption Secret was created automatically.&lt;/li&gt;
&lt;li&gt;A test Pod could write and read data.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;dmsetup&lt;/code&gt; showed a &lt;code&gt;crypt&lt;/code&gt; mapping.&lt;/li&gt;
&lt;li&gt;The ext4 filesystem was mounted above the encrypted mapping.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Separate checks on multiple volumes also found different hashes for their generated key material. That supports the intended per-PVC key separation.&lt;/p&gt;
&lt;p&gt;The distinction matters: declaring &lt;code&gt;encrypted: &quot;true&quot;&lt;/code&gt; is configuration; observing the actual dm-crypt device and readable filesystem is evidence that the storage path works.&lt;/p&gt;
&lt;h2 id=&quot;re-running-the-bootstrap-without-destroying-the-cluster&quot;&gt;Re-running the bootstrap without destroying the cluster&lt;/h2&gt;
&lt;p&gt;For me, one of the most useful tests was running the bootstrap again against the existing environment.&lt;/p&gt;
&lt;p&gt;The script distinguishes two modes:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Mode&lt;/th&gt;&lt;th&gt;Intended behavior&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Normal run&lt;/td&gt;&lt;td&gt;Reuse existing VMs and disks; reconcile the parts the script manages&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;RECREATE=1&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Remove managed VMs and disks, then recreate them; &lt;strong&gt;destroys their data&lt;/strong&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/cloud-agnostic-kubernetes-platform/bootstrap-flow.svg&quot; alt=&quot;Bootstrap and rerun behavior&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The normal rerun path preserves managed VM disks and the existing RKE2 CNI configuration. &lt;code&gt;RECREATE=1&lt;/code&gt; explicitly removes managed VM disks. A successful rerun therefore demonstrates preservation, &lt;strong&gt;not&lt;/strong&gt; the ability to recover after destruction.&lt;/p&gt;
&lt;p&gt;This is not just a convenience. A bootstrap that works once but destroys data on the second run is difficult to trust.&lt;/p&gt;
&lt;p&gt;The normal rerun smoke test passed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;All five RKE2 nodes remained &lt;code&gt;Ready&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Cilium agents remained &lt;code&gt;Running&lt;/code&gt;, without observed restarts.&lt;/li&gt;
&lt;li&gt;The test PVC stayed &lt;code&gt;Bound&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The previously written test data remained readable.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Rancher also reported the imported cluster as &lt;code&gt;Connected=True&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;What I did not verify:&lt;/strong&gt; a full destructive recreation followed by recovery of the original data. &lt;code&gt;RECREATE=1&lt;/code&gt; is an explicitly destructive path, not a disaster-recovery test. The verification record also does not establish every resource UID remained unchanged across the rerun.&lt;/p&gt;
&lt;h2 id=&quot;what-i-learned&quot;&gt;What I learned&lt;/h2&gt;
&lt;p&gt;The most time-consuming work was not writing the Kubernetes resources. It was making the entire environment predictable:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Networking comes first.&lt;/strong&gt; Libvirt NAT, host forwarding, SSH access, CNI selection, and ingress all interact. A working Kubernetes API does not prove applications are reachable.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Order matters.&lt;/strong&gt; Cilium settings must be in place for the first RKE2 bootstrap. Changing a live CNI should not be treated as a routine rerun operation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Storage encryption needs lifecycle rules.&lt;/strong&gt; Creating a unique key is straightforward. Preserving it, refusing accidental replacement, and recovering it after cluster loss are the difficult parts.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Idempotency is observable behavior.&lt;/strong&gt; A second successful script exit is not enough. I checked node readiness, Cilium state, PVC state, and whether existing data could still be read.&lt;/p&gt;
&lt;h2 id=&quot;what-v0-1-0-does-not-solve-yet&quot;&gt;What v0.1.0 does not solve yet&lt;/h2&gt;
&lt;p&gt;The current implementation is a reproducible &lt;strong&gt;local&lt;/strong&gt; foundation for a cloud-agnostic platform. It does not yet demonstrate deployments on multiple cloud providers.&lt;/p&gt;
&lt;p&gt;Several areas remain open:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Tenant provisioning:&lt;/strong&gt; automatic namespaces, RBAC, policies, quotas, and application stacks.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tenant object storage:&lt;/strong&gt; evaluating separate RustFS operator-managed instances and their bucket/credential isolation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Backup and recovery:&lt;/strong&gt; Longhorn backups to an S3-compatible target, plus independent backup and recovery of encryption Secrets.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Provider portability:&lt;/strong&gt; Terraform/Ansible implementations and provider-specific networking/storage integration.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Production hardening:&lt;/strong&gt; pinned images and release checksums, secure credential handling, verified SSH host keys, external key management, and a production control-plane topology.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The existing RustFS service runs in a separate k3s cluster on the same physical host. It may be useful as an initial S3-compatible Longhorn backup target, but &lt;strong&gt;it is not an off-host disaster-recovery destination&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;There is another important topology limitation: the tested RKE2 cluster has &lt;strong&gt;two server/etcd nodes&lt;/strong&gt;. This is the observed lab configuration, not a recommended production HA quorum design.&lt;/p&gt;
&lt;h2 id=&quot;next-steps&quot;&gt;Next steps&lt;/h2&gt;
&lt;p&gt;I froze &lt;strong&gt;Cluster Bootstrap v0.1.0&lt;/strong&gt; as the baseline for this experiment. The next step is to connect Longhorn backup to RustFS, save volume encryption keys independently, and prove that an encrypted volume can be restored with its original data.&lt;/p&gt;
&lt;p&gt;After that, I want to move toward repeatable tenant provisioning and replace the host-specific VM provisioning with provider-aware infrastructure automation.&lt;/p&gt;
&lt;p&gt;The point of this first version was not to build the whole platform in one script. It was to establish a working, testable base—and to find out which assumptions break when I deploy it again.&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;&lt;em&gt;Technical basis: the frozen &lt;code&gt;bootstrap.sh&lt;/code&gt;, &lt;code&gt;README.md&lt;/code&gt;, and &lt;code&gt;VERIFICATION.md&lt;/code&gt; from Cluster Bootstrap v0.1.0 (October 26, 2026). The verification results are observations from the running lab, not a fresh test performed for this article.&lt;/em&gt;&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Why My AI Agent Stopped After 4–5 Iterations</title>
        <published>2026-10-23T00:00:00+00:00</published>
        <updated>2026-10-23T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/"/>
        <id>https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/hero.png&quot; alt=&quot;Abstract illustration of an AI agent stopping at a context boundary&quot; /&gt;&lt;/p&gt;
&lt;p&gt;While working on &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/agent-loop&quot;&gt;Agent Loop&lt;/a&gt;, my Python runtime for locally hosted LLMs, I ran into a frustrating problem: the agent would start a task, make a few tool calls, and then stop after four or five iterations.&lt;/p&gt;
&lt;p&gt;The model&#39;s response contained &lt;code&gt;finish_reason: length&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;My first suspicion was the model or the inference server. I was running a local setup, after all, and long agent sessions put quite different demands on an LLM than a single chat response. But the problem turned out to be in the runtime&#39;s token configuration.&lt;/p&gt;
&lt;h2 id=&quot;the-setup&quot;&gt;The setup&lt;/h2&gt;
&lt;p&gt;I was testing with GLM-4.7-Flash through a local OpenAI-compatible Chat Completions endpoint. Agent Loop handles the execution cycle: it sends the conversation to the model, processes tool calls, appends tool results, and continues until the task is finished or the runtime stops.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/agent-token-flow.svg&quot; alt=&quot;Agent execution loop and token limits&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The important detail is that an agent task is not one model request. Every iteration sends another request, often with a larger conversation history and additional tool output.&lt;/p&gt;
&lt;p&gt;That means the runtime has to manage more than one token limit.&lt;/p&gt;
&lt;h2 id=&quot;the-mistake&quot;&gt;The mistake&lt;/h2&gt;
&lt;p&gt;In my initial implementation, I effectively used the overall conversation budget as the maximum output allowance for each individual request:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Simplified illustration of the old configuration&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;max_tokens&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 131072&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The value &lt;code&gt;131072&lt;/code&gt; belonged to the session-level budget. It should not have been passed directly as the generation limit for every Chat Completions request.&lt;/p&gt;
&lt;p&gt;A high &lt;code&gt;max_tokens&lt;/code&gt; setting does &lt;strong&gt;not&lt;/strong&gt; mean the model will generate that many tokens. It specifies an upper bound. But the inference server may validate or reserve room for that allowance against the available context, and an unrealistic limit makes context handling harder to reason about.&lt;/p&gt;
&lt;p&gt;I had mixed together three separate concepts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Model context window&lt;/strong&gt; — the maximum number of tokens the model/backend can handle in a request, including input and generated output.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Per-request output limit&lt;/strong&gt; — the maximum number of tokens the model is allowed to generate for one response.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Conversation budget&lt;/strong&gt; — the runtime&#39;s application-level limit for accumulated conversation state, used to decide when to compact or stop.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/token-budgets.svg&quot; alt=&quot;Three independent token budgets&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The model&#39;s context capacity and the application&#39;s conversation budget can have the same configured number, but they still serve different purposes.&lt;/p&gt;
&lt;h2 id=&quot;fixing-the-configuration&quot;&gt;Fixing the configuration&lt;/h2&gt;
&lt;p&gt;I separated the settings instead of reusing one number everywhere:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;MODEL_CONTEXT_WINDOW&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 131072&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;REQUEST_MAX_TOKENS&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8192&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;MAX_TOKENS&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 131072&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;COMPACT_THRESHOLD&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0.8&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;These are the values from the debugging setup, not universal defaults for every model.&lt;/p&gt;
&lt;p&gt;The API request now uses the per-request limit:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Simplified request configuration&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;max_tokens&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; =&lt;/span&gt;&lt;span&gt; config&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;REQUEST_MAX_TOKENS&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Meanwhile, the runtime tracks the conversation budget independently and can compact the history as it grows.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/why-my-ai-agent-stopped-after-4-5-iterations/context-compaction.svg&quot; alt=&quot;Conversation compaction and budget decisions&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This separation matters even if the model supports a large context window. A request still needs enough room for its input, generated response, and any backend-specific overhead. Compaction helps reduce accumulated history; it does not magically increase the context window configured in the inference server.&lt;/p&gt;
&lt;h2 id=&quot;what-the-error-actually-told-me&quot;&gt;What the error actually told me&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;finish_reason: length&lt;/code&gt; was a useful clue, but not a complete diagnosis.&lt;/p&gt;
&lt;p&gt;It tells you that generation stopped because of a length-related limit. By itself, it does not tell you whether the problem is the requested output cap, the available context, the server&#39;s configuration, or the way your application builds requests.&lt;/p&gt;
&lt;p&gt;The useful debugging step was to inspect the outgoing request parameters, rather than assume that the model was simply unable to continue.&lt;/p&gt;
&lt;p&gt;For an agent loop, I would now check:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The actual context window configured in the inference backend.&lt;/li&gt;
&lt;li&gt;The size of the conversation and tool results sent with each iteration.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;max_tokens&lt;/code&gt; value attached to each model request.&lt;/li&gt;
&lt;li&gt;The runtime&#39;s own conversation-budget and compaction decisions.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Those numbers need to agree with each other, but they should not be represented by one shared setting.&lt;/p&gt;
&lt;h2 id=&quot;after-the-change&quot;&gt;After the change&lt;/h2&gt;
&lt;p&gt;Before the fix, my test runs commonly stopped after about &lt;strong&gt;4–5 iterations&lt;/strong&gt;. With the limits separated, I observed runs reaching &lt;strong&gt;20–50+ iterations&lt;/strong&gt; in some development scenarios.&lt;/p&gt;
&lt;p&gt;This is not a benchmark or a promise that the agent can complete any 50-step task. The result depends on the prompt, model, tool output sizes, context configuration, and inference backend. It does show that the earlier stopping point was not an inherent four-iteration limit of the model.&lt;/p&gt;
&lt;h2 id=&quot;what-i-took-away&quot;&gt;What I took away&lt;/h2&gt;
&lt;p&gt;I started by debugging what looked like an LLM problem and ended up fixing an orchestration problem.&lt;/p&gt;
&lt;p&gt;The lesson was simple: &lt;strong&gt;context capacity, per-request generation limits, and conversation lifetime are different things&lt;/strong&gt;. In an agent runtime, they should be configured and monitored separately.&lt;/p&gt;
&lt;p&gt;This is one of those small implementation details that is easy to overlook in a short chat demo, but becomes obvious once the model is expected to work through a longer sequence of tool calls.&lt;/p&gt;
&lt;p&gt;The code is available in &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/agent-loop&quot;&gt;Agent Loop on GitHub&lt;/a&gt;. For an overview of the runtime itself, see &lt;a href=&quot;/blog/agent-loop-local-llm-agent-framework/&quot;&gt;Building Agent Loop: A Local-LLM Agent Runtime in Python&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Building Agent Loop: A Local-LLM Agent Runtime in Python</title>
        <published>2026-10-19T00:00:00+00:00</published>
        <updated>2026-10-19T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/agent-loop-local-llm-agent-framework/"/>
        <id>https://iamavk.info/blog/agent-loop-local-llm-agent-framework/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/agent-loop-local-llm-agent-framework/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/hero.png&quot; alt=&quot;Abstract illustration of a local LLM agent runtime and execution loop&quot; /&gt;&lt;/p&gt;
&lt;p&gt;AI coding tools are useful, but I wanted to understand and control the part between a model response and a real action: &lt;strong&gt;the agent loop itself&lt;/strong&gt;. What decides whether to call a tool? How are tool results returned to the model? What happens when a task takes dozens of iterations, and how can a local model keep working without exhausting its context window?&lt;/p&gt;
&lt;p&gt;That experiment became &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/agent-loop&quot;&gt;Agent Loop&lt;/a&gt;, an open-source Python framework for running tool-using agents against locally hosted language models. It supports configurable agent personas, skills, shell and filesystem tools, MCP (Model Context Protocol) integrations, session persistence, response caching, and interactive or one-shot execution.&lt;/p&gt;
&lt;p&gt;The interesting part was not writing another chat client. It was building a small, understandable runtime and then debugging the failures that appear only when a model starts taking actions repeatedly.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/architecture.svg&quot; alt=&quot;Agent Loop architecture: CLI, loop, model API, tool dispatch and context management&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;why-build-another-agent-runtime&quot;&gt;Why build another agent runtime?&lt;/h2&gt;
&lt;p&gt;I had already been experimenting with local models in software-development workflows, including &lt;a href=&quot;/blog/spec-driven-ai-development-bgpd-simple/&quot;&gt;spec-driven development with bgpd-simple&lt;/a&gt;. Those experiments made me interested in the execution layer rather than only the model or the prompt.&lt;/p&gt;
&lt;p&gt;An agent is more than a chat completion. It needs to maintain state, expose tools in a format the model understands, validate and dispatch tool calls, feed results back into the conversation, and decide when to stop. For long-running tasks it also needs to manage a finite context budget.&lt;/p&gt;
&lt;p&gt;My goals for Agent Loop were deliberately practical:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Run primarily against &lt;strong&gt;local inference servers&lt;/strong&gt;, without requiring an Anthropic or OpenAI cloud account.&lt;/li&gt;
&lt;li&gt;Keep the &lt;strong&gt;agent loop, tools, and model-serving backend&lt;/strong&gt; separate.&lt;/li&gt;
&lt;li&gt;Define reusable &lt;strong&gt;agents&lt;/strong&gt;, &lt;strong&gt;skills&lt;/strong&gt;, and project-specific instructions.&lt;/li&gt;
&lt;li&gt;Connect external capabilities through &lt;strong&gt;MCP&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Support both an interactive terminal session and a one-shot command.&lt;/li&gt;
&lt;li&gt;Make token usage and context compaction explicit rather than hiding them behind an SDK.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is a personal engineering project, not a claim to replace established agent frameworks. Its value to me is that I can inspect and change the complete control flow.&lt;/p&gt;
&lt;h2 id=&quot;the-execution-loop&quot;&gt;The execution loop&lt;/h2&gt;
&lt;p&gt;At a high level, each iteration builds a model request from the current conversation and available tools. The model can return a normal answer or one or more tool calls. When tools are requested, the runtime dispatches them, captures their results, appends those results to the conversation, and calls the model again.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/loop.svg&quot; alt=&quot;Agent execution loop: goal, request, model response, tools, context update and final answer&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/tool-call-sequence.svg&quot; alt=&quot;Agent Loop tool-call sequence: model request, dispatch, tool result, and next model turn&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The implementation separates these concerns into modules such as &lt;code&gt;src/loop.py&lt;/code&gt;, &lt;code&gt;src/dispatch.py&lt;/code&gt;, &lt;code&gt;src/session.py&lt;/code&gt;, &lt;code&gt;src/compaction.py&lt;/code&gt;, and &lt;code&gt;src/tools/&lt;/code&gt;. This makes it easier to distinguish an inference problem from a tool-dispatch problem or a context-management problem.&lt;/p&gt;
&lt;p&gt;There are two primary ways to use the CLI:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Execute one goal&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; run&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Investigate the failing tests&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Continue an interactive session&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; repl&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Named agents and skills let the same runtime handle different tasks without giving every task the same system prompt or tool permissions:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; investigator&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Find the root cause of this timeout&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; code-reviewer&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Review src/api.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; skill&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; data-analyst&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Analyze data/metrics.csv&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The repository includes agents such as &lt;code&gt;coder&lt;/code&gt;, &lt;code&gt;investigator&lt;/code&gt;, &lt;code&gt;researcher&lt;/code&gt;, &lt;code&gt;code-reviewer&lt;/code&gt;, and &lt;code&gt;local-assistant&lt;/code&gt;. Skills provide narrower capability packages. &lt;code&gt;AGENTS.md&lt;/code&gt; can supply repository-specific rules.&lt;/p&gt;
&lt;h2 id=&quot;local-inference-api-protocol-is-not-the-model&quot;&gt;Local inference: API protocol is not the model&lt;/h2&gt;
&lt;p&gt;One source of confusion in local AI tooling is the word &lt;em&gt;compatibility&lt;/em&gt;. A server, API protocol, and model are three different components.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/backends.svg&quot; alt=&quot;Agent Loop uses an OpenAI-compatible API with either LM Studio or llama.cpp and a local model&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Agent Loop currently sends OpenAI-compatible streaming chat-completions requests&lt;/strong&gt;, including tool definitions, to the endpoint configured as &lt;code&gt;LM_STUDIO_URL&lt;/code&gt;. That variable name is historical: it also points to other compatible inference backends.&lt;/p&gt;
&lt;p&gt;My development setup uses &lt;strong&gt;GLM-4.7-Flash&lt;/strong&gt; with &lt;strong&gt;LM Studio&lt;/strong&gt;. The repository also includes an example configuration for &lt;strong&gt;llama.cpp&lt;/strong&gt; and &lt;code&gt;llama-server&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;llama-server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /path/to/GLM-4.7-Flash.gguf&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 131072&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-jinja&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8080&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.py&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; run&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Inspect this repository&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-config&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent_loop.llamacpp.yaml&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The important qualification is that &lt;em&gt;OpenAI-compatible&lt;/em&gt; does not mean every endpoint or model will work automatically. Agent Loop expects a particular streaming response and tool-call structure. The inference server, model, and chat template must produce that structure consistently.&lt;/p&gt;
&lt;p&gt;LM Studio may also expose an Anthropic/Claude-compatible API, but &lt;strong&gt;Agent Loop itself does not directly implement the Anthropic Messages API&lt;/strong&gt;. It does not need a hosted Claude model to run.&lt;/p&gt;
&lt;h2 id=&quot;mcp-extending-the-runtime-beyond-built-in-tools&quot;&gt;MCP: extending the runtime beyond built-in tools&lt;/h2&gt;
&lt;p&gt;Built-in shell and filesystem operations are useful for local development. MCP provides another integration boundary: external servers can expose tools without requiring each integration to be hard-coded into the agent loop.&lt;/p&gt;
&lt;p&gt;The project uses &lt;code&gt;mcp_servers.json&lt;/code&gt; to describe MCP servers, including npm-based tools such as Context7. Those npm-based integrations require Node.js, but the Python agent runtime itself does not.&lt;/p&gt;
&lt;p&gt;This separation matters operationally. The model sees a set of tool schemas; the runtime controls how the requested tools are executed. Tool access is not equivalent to permission to run arbitrary commands. Agent Loop exposes allowlists and an explicit &lt;code&gt;--allow&lt;/code&gt; option, which skips approval prompts. I use broad access only in trusted environments, especially when shell execution or filesystem writes are enabled.&lt;/p&gt;
&lt;h2 id=&quot;the-failure-that-made-token-management-important&quot;&gt;The failure that made token management important&lt;/h2&gt;
&lt;p&gt;During development I encountered a frustrating behavior: the agent would stop after approximately &lt;strong&gt;four or five iterations&lt;/strong&gt;, reporting &lt;code&gt;finish_reason: length&lt;/code&gt;. The model could complete short tasks, but longer workflows terminated much too early.&lt;/p&gt;
&lt;p&gt;The underlying configuration mixed together two different limits. The total conversation budget was set to &lt;strong&gt;131,072 tokens&lt;/strong&gt;, and that same value had been used as the &lt;strong&gt;per-request maximum output&lt;/strong&gt;. A session-wide budget is not a sensible default for a single completion. This did not mean the model literally generated 131,072 tokens every time; it meant the request advertised an excessively large output allowance relative to the available context.&lt;/p&gt;
&lt;p&gt;The fix was to separate the relevant controls:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/agent-loop-local-llm-agent-framework/budget.svg&quot; alt=&quot;Four token-management concepts: context window, per-request output limit, conversation budget, and compaction&quot; /&gt;&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Setting&lt;/th&gt;&lt;th&gt;Purpose&lt;/th&gt;&lt;th style=&quot;text-align: right&quot;&gt;Development configuration&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;MODEL_CONTEXT_WINDOW&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Actual model context capacity&lt;/td&gt;&lt;td style=&quot;text-align: right&quot;&gt;131,072&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;REQUEST_MAX_TOKENS&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Maximum output for one request&lt;/td&gt;&lt;td style=&quot;text-align: right&quot;&gt;8,192&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;MAX_TOKENS&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Total conversation budget&lt;/td&gt;&lt;td style=&quot;text-align: right&quot;&gt;131,072&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;COMPACT_THRESHOLD&lt;/code&gt;&lt;/td&gt;&lt;td&gt;When to compact history&lt;/td&gt;&lt;td style=&quot;text-align: right&quot;&gt;0.8&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;With the output limit separated from the session budget, development tests observed &lt;strong&gt;20–50+ iterations&lt;/strong&gt; rather than stopping after four or five. These are observations from the development setup, not guaranteed iteration counts or a controlled model benchmark.&lt;/p&gt;
&lt;h3 id=&quot;compaction-is-a-fallback-not-unlimited-memory&quot;&gt;Compaction is a fallback, not unlimited memory&lt;/h3&gt;
&lt;p&gt;As conversation history grows, the runtime tracks its approximate token usage. At the configured threshold, it tries to summarize older messages while preserving the system prompt and recent messages. If summarization fails, a hard-truncation fallback keeps the most recent context.&lt;/p&gt;
&lt;p&gt;That design has trade-offs. A summary can lose details, while truncation can lose more. Neither creates an infinite context window. But both are preferable to treating every accumulated tool result as permanently necessary.&lt;/p&gt;
&lt;p&gt;It is also important to distinguish &lt;strong&gt;context-budget pressure&lt;/strong&gt; from &lt;strong&gt;RAM/VRAM exhaustion&lt;/strong&gt;. Reducing a per-request output limit may help the former; it will not necessarily make an oversized model fit into GPU memory.&lt;/p&gt;
&lt;h2 id=&quot;a-few-engineering-lessons&quot;&gt;A few engineering lessons&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;1. Protocol compatibility must be tested at the tool-call boundary.&lt;/strong&gt; A successful plain-text completion does not prove that streaming tool calls will be parsed correctly.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Agent permissions are part of the runtime design.&lt;/strong&gt; A tool schema tells the model what it &lt;em&gt;can request&lt;/em&gt;; allowlists and approval behavior determine what the runtime &lt;em&gt;will execute&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;3. Long-running tasks expose configuration mistakes quickly.&lt;/strong&gt; The &lt;code&gt;finish_reason: length&lt;/code&gt; issue was not obvious from a single request. It appeared when the loop accumulated enough history to make token budgeting matter.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;4. Local inference changes the constraints, not the fundamentals.&lt;/strong&gt; I can control the serving stack and keep prompts local, but I still need reliable tool calling, explicit context limits, and sensible failure handling.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;5. Small modules make debugging easier.&lt;/strong&gt; Separating the CLI, agent loop, dispatch, sessions, MCP integration, and compaction made it easier to isolate the source of a failure.&lt;/p&gt;
&lt;h2 id=&quot;try-agent-loop&quot;&gt;Try Agent Loop&lt;/h2&gt;
&lt;p&gt;The source code, configuration examples, agent definitions, and setup instructions are available on GitHub:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/agent-loop&quot;&gt;github.com/avkghost/agent-loop&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;git&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; clone&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; https://github.com/avkghost/agent-loop.git&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;cd&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; agent-loop&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;pip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; install&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;r&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; requirements.txt&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Choose a compatible local inference server, configure its endpoint and model, and start with a small read-only task before enabling broader tool access. The project is MIT-licensed.&lt;/p&gt;
&lt;p&gt;Agent Loop remains an experimental engineering project. I built it to understand and improve the reliability of local tool-using agents—and to keep that runtime understandable enough to debug when something goes wrong.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Spec-Driven AI Development: Building bgpd-simple with Spec Kit and Local AI Coding Agents</title>
        <published>2026-10-15T00:00:00+00:00</published>
        <updated>2026-10-15T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/"/>
        <id>https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/hero.png&quot; alt=&quot;Abstract illustration of specification-driven AI software development&quot; /&gt;&lt;/p&gt;
&lt;p&gt;AI coding becomes much more useful when the model is not asked to invent both the requirements and the implementation at the same time.&lt;/p&gt;
&lt;p&gt;That was one of the main lessons from working on &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple&quot;&gt;&lt;code&gt;bgpd-simple&lt;/code&gt;&lt;/a&gt;, a compact routing daemon written in C with BGP, OSPF, and MPLS/LDP support. The project is small enough to understand, but protocol work is unforgiving: a change can look perfectly reasonable in C and still be wrong because it violates one sentence in an RFC, serializes one field incorrectly, or breaks a state transition that only appears during reconnect or withdrawal.&lt;/p&gt;
&lt;p&gt;For this project I used &lt;strong&gt;Spec Kit&lt;/strong&gt; as the workflow layer, &lt;strong&gt;OpenCode&lt;/strong&gt; as the coding agent, and &lt;strong&gt;local models served by llama.cpp&lt;/strong&gt;. I started with &lt;strong&gt;Qwen 3.5&lt;/strong&gt; and &lt;strong&gt;GLM-4.7&lt;/strong&gt;, then experimented with newer models, including &lt;strong&gt;Qwen 3.6 27B&lt;/strong&gt;. The important part was not the particular model. The important part was moving as much engineering truth as possible out of the prompt and into versioned artifacts: specifications, RFC references, plans, task lists, tests, compliance matrices, and verification reports.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/sdd-workflow.svg&quot; alt=&quot;Spec-driven development loop&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;why-a-routing-daemon-is-a-good-sdd-test-case&quot;&gt;Why a routing daemon is a good SDD test case&lt;/h2&gt;
&lt;p&gt;A routing daemon is almost the opposite of a “vibe coding” application.&lt;/p&gt;
&lt;p&gt;The external behavior is already specified. BGP is defined by documents such as &lt;a rel=&quot;external&quot; href=&quot;https://datatracker.ietf.org/doc/rfc4271/&quot;&gt;RFC 4271&lt;/a&gt;, multiprotocol BGP by &lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc4760.html&quot;&gt;RFC 4760&lt;/a&gt;, OSPFv2 by &lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc2328.html&quot;&gt;RFC 2328&lt;/a&gt;, OSPFv3 by &lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5340.html&quot;&gt;RFC 5340&lt;/a&gt;, and LDP by &lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5036.html&quot;&gt;RFC 5036&lt;/a&gt;. The implementation may be original, but the observable protocol behavior cannot be arbitrary.&lt;/p&gt;
&lt;p&gt;That makes the project a useful environment for spec-driven development because there are several independent sources of truth:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;protocol RFCs;&lt;/li&gt;
&lt;li&gt;the project configuration and CLI vocabulary;&lt;/li&gt;
&lt;li&gt;internal architecture and invariants;&lt;/li&gt;
&lt;li&gt;regression tests;&lt;/li&gt;
&lt;li&gt;interoperability behavior;&lt;/li&gt;
&lt;li&gt;explicit release/compliance gates.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The repository reflects this. Documentation is split into protocol domains and lifecycle-oriented material: BGP, OSPF, MPLS/LDP, shared architecture, troubleshooting, compliance matrices, and project reports. The release documentation also requires compliance claims to point to executable evidence rather than to prose alone.&lt;/p&gt;
&lt;p&gt;In other words, the specification is not one large Markdown file. It is a graph of constraints.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/evidence-loop.svg&quot; alt=&quot;Specification and evidence graph&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-stack-spec-kit-opencode-or-claude-code-llama-cpp-or-lm-studio&quot;&gt;The stack: Spec Kit, OpenCode or Claude Code, llama.cpp or LM Studio&lt;/h2&gt;
&lt;p&gt;The roles were deliberately separated. The primary configuration below is &lt;strong&gt;OpenCode + llama.cpp&lt;/strong&gt;; &lt;strong&gt;Claude Code&lt;/strong&gt; is an alternative coding agent, while &lt;strong&gt;LM Studio&lt;/strong&gt; is an alternative local inference server. These are interchangeable components, not a claim that every combination was used for every bgpd-simple commit.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Spec Kit&lt;/strong&gt; provided the structured development workflow. It turns an idea into artifacts such as a specification, technical plan, task list, and implementation/convergence steps. &lt;strong&gt;OpenCode&lt;/strong&gt; acted as the coding driver: reading the repository, editing files, running builds and tests, and iterating on failures. &lt;strong&gt;llama.cpp&lt;/strong&gt; supplied an OpenAI-compatible local HTTP endpoint. &lt;strong&gt;Qwen 3.5&lt;/strong&gt; and &lt;strong&gt;GLM-4.7&lt;/strong&gt; were my starting points, followed by experiments with newer models, including &lt;strong&gt;Qwen 3.6 27B&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/tooling-stack.svg&quot; alt=&quot;Interchangeable agents and local inference servers&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This distinction matters. Spec Kit is not the model and it is not the coding agent. It is the process layer. OpenCode &lt;strong&gt;or Claude Code&lt;/strong&gt; drives repository edits and tests; llama.cpp &lt;strong&gt;or LM Studio&lt;/strong&gt; serves a model locally. The model is replaceable as long as it can follow the process, reason over the repository, and use tools reliably.&lt;/p&gt;
&lt;p&gt;Current Spec Kit releases list &lt;code&gt;opencode&lt;/code&gt; as a supported integration and install OpenCode command files under &lt;code&gt;.opencode/commands&lt;/code&gt;. Current documentation often shows skill names such as &lt;code&gt;/speckit-specify&lt;/code&gt;, &lt;code&gt;/speckit-plan&lt;/code&gt;, and &lt;code&gt;/speckit-tasks&lt;/code&gt;. During this work I used the dotted command form commonly seen in earlier Spec Kit workflows — &lt;code&gt;/speckit.specify&lt;/code&gt;, &lt;code&gt;/speckit.plan&lt;/code&gt;, and &lt;code&gt;/speckit.tasks&lt;/code&gt;. The exact slash-command spelling is version/integration dependent; the workflow boundary is the important part.&lt;/p&gt;
&lt;h2 id=&quot;from-qwen-3-5-and-glm-4-7-to-newer-local-models&quot;&gt;From Qwen 3.5 and GLM-4.7 to newer local models&lt;/h2&gt;
&lt;p&gt;I did not start this project with Qwen 3.6 27B. My early experiments used &lt;strong&gt;Qwen 3.5&lt;/strong&gt; and &lt;strong&gt;GLM-4.7&lt;/strong&gt;. As newer model releases became available, I tried them with the same general setup: a local &lt;code&gt;llama.cpp&lt;/code&gt; inference server exposed through OpenCode, with Spec Kit supplying the development workflow. &lt;strong&gt;Qwen 3.6 27B&lt;/strong&gt; is one of the newer configurations discussed below, not the only model used over the lifetime of the project.&lt;/p&gt;
&lt;p&gt;This model progression is useful context for the SDD approach. Rather than tie protocol requirements to a particular model or chat history, I could keep the RFC references, plans, test commands, and acceptance checks in the repository while changing the inference backend. I am not presenting controlled model benchmarks here: the important observation is the portability of the &lt;em&gt;workflow&lt;/em&gt;, not an unmeasured claim that one model outperformed another.&lt;/p&gt;
&lt;h3 id=&quot;the-hardware-behind-the-local-experiments&quot;&gt;The hardware behind the local experiments&lt;/h3&gt;
&lt;p&gt;My &lt;strong&gt;main setup&lt;/strong&gt; was a server/workstation built around a &lt;strong&gt;Supermicro H11SSL-C motherboard&lt;/strong&gt;, an &lt;strong&gt;AMD EPYC 7551 processor&lt;/strong&gt;, and &lt;strong&gt;256 GB RAM&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;For an &lt;strong&gt;alternative setup&lt;/strong&gt;, I used a &lt;strong&gt;Mougol X99&lt;/strong&gt; motherboard and later moved to a &lt;strong&gt;Machinist X99 D8 Max&lt;/strong&gt; board, with &lt;strong&gt;two Intel Xeon E5-2680 processors&lt;/strong&gt; and &lt;strong&gt;64 GB RAM&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The GPUs available for the experiments were:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;PNY GeForce RTX 5060 Ti OC — 16 GB VRAM&lt;/strong&gt;;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Gainward GeForce RTX 5070 Ti Phoenix-S — 16 GB VRAM&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/local-ai-hardware.svg&quot; alt=&quot;Local AI development hardware: main EPYC server, alternative dual-Xeon X99 machine, and GPUs&quot; /&gt;&lt;/p&gt;
&lt;p&gt;On the &lt;strong&gt;Machinist X99 D8 Max&lt;/strong&gt;, PCIe/NUMA topology matters when distributing inference work across GPUs. PCIe slots may be attached to different CPU sockets, and communication between GPUs on opposite sockets can cross the CPU-to-CPU interconnect. In my setup, that became a bottleneck. &lt;strong&gt;For multi-GPU workloads with frequent cross-GPU transfers, I place both cards in PCIe slots attached to the same CPU&lt;/strong&gt; whenever the board&#39;s slot wiring, physical clearance, and available PCIe lanes permit it. This avoids the extra inter-socket hop; it does not eliminate normal PCIe transfer costs. Check the actual mapping with &lt;code&gt;lspci -tv&lt;/code&gt; and &lt;code&gt;nvidia-smi topo -m&lt;/code&gt; rather than assuming that adjacent slots share a CPU. The best choice still depends on the model split and the transfer pattern.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/dual-socket-gpu-topology.svg&quot; alt=&quot;Dual-socket PCIe topology: GPU placement across CPUs versus within one CPU&amp;#39;s PCIe domain&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The &lt;strong&gt;H11SSL-C&#39;s revision supports EPYC 7002 processors&lt;/strong&gt;, so an &lt;strong&gt;EPYC 7502&lt;/strong&gt; upgrade would also be possible. That upgrade deserves a separate article; the experiments described here used the &lt;strong&gt;EPYC 7551&lt;/strong&gt;. See &lt;a rel=&quot;external&quot; href=&quot;https://www.supermicro.com/en/support/faqs/faq.php?faq=30897&quot;&gt;Supermicro&#39;s official EPYC 7002 support FAQ&lt;/a&gt; for the revision requirement.&lt;/p&gt;
&lt;p&gt;For models that do not fit entirely into a single 16 GB VRAM allocation, &lt;code&gt;llama.cpp&lt;/code&gt; can use quantized GGUF weights and appropriate CPU/GPU offloading; the practical fit and speed depend on the quantization, available RAM, context length, and the actual GPU configuration. The hardware list above describes my test equipment, not a claim that both GPUs were always installed together or that every model ran fully in VRAM.&lt;/p&gt;
&lt;p&gt;This is why a local OpenAI-compatible endpoint was so valuable: OpenCode&#39;s connection remained broadly the same even as I changed machines, models, and inference settings.&lt;/p&gt;
&lt;h2 id=&quot;adopting-spec-kit-in-an-existing-repository&quot;&gt;Adopting Spec Kit in an existing repository&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;bgpd-simple&lt;/code&gt; was not a greenfield “generate an app” exercise. For an existing repository, I prefer to initialize Spec Kit in place from a clean, reviewable Git state:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;uv&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; tool&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; install&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; specify-cli&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;cd&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; bgpd-simple&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;git&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; status&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;specify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; init&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-here&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-force&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-integration&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; opencode&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For a repository that already matters, &lt;code&gt;--force&lt;/code&gt; should not mean “overwrite without looking.” It means the generated integration files are applied to an existing tree, so I want the pre-init state committed or stashed and I review the diff immediately afterward.&lt;/p&gt;
&lt;p&gt;The resulting separation is useful:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;.specify/              shared Spec Kit workflow data&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;.opencode/commands/    OpenCode integration commands&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;specs/                 feature specifications / plans / tasks&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;src/, include/         implementation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;tests/                 executable behavior&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;docs/                  architecture, RFC compliance, reports&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For protocol work I also keep project-level rules explicit. The repository&#39;s &lt;code&gt;docs/meta/WO.md&lt;/code&gt; captures several of them: plan first for non-trivial work, minimize the impact of changes, do not accept temporary fixes when the root cause is knowable, and never mark a task done without verification. Those rules are exactly the kind of constraints I want an agent to inherit before it starts editing C.&lt;/p&gt;
&lt;h2 id=&quot;a-practical-spec-kit-workflow&quot;&gt;A practical Spec Kit workflow&lt;/h2&gt;
&lt;p&gt;For a non-trivial protocol change, my workflow is roughly:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;constitution / project rules&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;specify the externally visible behavior&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;clarify ambiguous protocol or compatibility points&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;plan the implementation boundaries&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;turn the plan into checkable tasks&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;analyze the plan against the specification&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;implement with OpenCode&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;run targeted tests + full regression gate&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;update compliance/evidence documents&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        ↓&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;converge only when code, tests and docs agree&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In command form, depending on the Spec Kit version:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.constitution   or /speckit-constitution&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.specify        or /speckit-specify&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.clarify        or /speckit-clarify&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.plan           or /speckit-plan&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.tasks          or /speckit-tasks&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.analyze        or /speckit-analyze&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.implement      or /speckit-implement&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;/speckit.converge       or /speckit-converge&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I do not necessarily use every phase for a tiny bug. The value comes from making the amount of process proportional to the risk.&lt;/p&gt;
&lt;p&gt;For example, changing a CLI string is not the same as changing BGP UPDATE encoding. For the second case I want the plan to mention the exact RFC section, affected parser/encoder path, malformed-input behavior, regression test, and release evidence update before the agent starts coding.&lt;/p&gt;
&lt;h2 id=&quot;rfcs-as-executable-requirements&quot;&gt;RFCs as executable requirements&lt;/h2&gt;
&lt;p&gt;The repository&#39;s compliance summary is more useful to an AI agent than a generic instruction such as “implement BGP correctly.” It maps protocol areas to RFCs, scope, gate status, and required evidence.&lt;/p&gt;
&lt;p&gt;Some of the core references used by the project are:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Domain&lt;/th&gt;&lt;th&gt;RFC&lt;/th&gt;&lt;th&gt;Purpose in the project&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://datatracker.ietf.org/doc/rfc4271/&quot;&gt;RFC 4271&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Core BGP-4 protocol&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc4760.html&quot;&gt;RFC 4760&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Multiprotocol extensions&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc6793.html&quot;&gt;RFC 6793&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Four-octet AS numbers&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc4456.html&quot;&gt;RFC 4456&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Route reflection&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5065.html&quot;&gt;RFC 5065&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Confederations&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc8654.html&quot;&gt;RFC 8654&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Extended messages&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc9774.html&quot;&gt;RFC 9774&lt;/a&gt;&lt;/td&gt;&lt;td&gt;AS_PATH segment handling updates&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc2328.html&quot;&gt;RFC 2328&lt;/a&gt;&lt;/td&gt;&lt;td&gt;OSPFv2&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5340.html&quot;&gt;RFC 5340&lt;/a&gt;&lt;/td&gt;&lt;td&gt;OSPF for IPv6&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc3623.html&quot;&gt;RFC 3623&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Graceful restart&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5187.html&quot;&gt;RFC 5187&lt;/a&gt;&lt;/td&gt;&lt;td&gt;OSPFv3 graceful restart&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5709.html&quot;&gt;RFC 5709&lt;/a&gt;&lt;/td&gt;&lt;td&gt;HMAC-SHA authentication&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;OSPF&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc4552.html&quot;&gt;RFC 4552&lt;/a&gt;&lt;/td&gt;&lt;td&gt;OSPFv3 authentication/confidentiality&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;MPLS/LDP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5036.html&quot;&gt;RFC 5036&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Label Distribution Protocol&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;MPLS&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc3032.html&quot;&gt;RFC 3032&lt;/a&gt;&lt;/td&gt;&lt;td&gt;MPLS label stack encoding&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;LDP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc6720.html&quot;&gt;RFC 6720&lt;/a&gt;&lt;/td&gt;&lt;td&gt;LDP capability negotiation&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;LDP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc5918.html&quot;&gt;RFC 5918&lt;/a&gt;&lt;/td&gt;&lt;td&gt;LDP IPv6 extensions&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;MPLS PW&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc4447.html&quot;&gt;RFC 4447&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Pseudowire signaling&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;LDP/IGP&lt;/td&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.rfc-editor.org/rfc/rfc7552.html&quot;&gt;RFC 7552&lt;/a&gt;&lt;/td&gt;&lt;td&gt;LDP/IGP synchronization&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;The useful pattern is not “put RFC links in a README.” It is to turn them into acceptance criteria.&lt;/p&gt;
&lt;p&gt;A specification for an encoder change might say:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Requirement:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- UPDATE encoding must follow RFC 4271 section 4.3.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- Empty-path behavior must be deterministic and interoperable.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- Malformed input must be rejected without corrupting peer/session state.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- Existing IPv4/IPv6 and AS4 tests must remain green.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- Add a focused regression reproducing the previously observed failure.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;- Update the BGP compliance matrix if the supported behavior changes.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is far harder for an agent to “interpret creatively” than “fix AS_PATH handling.”&lt;/p&gt;
&lt;h2 id=&quot;what-the-commit-history-shows&quot;&gt;What the commit history shows&lt;/h2&gt;
&lt;p&gt;The repository&#39;s own implementation ledger preserves a useful sequence of phase checkpoints from the March 7, 2026 stabilization work.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/commit-timeline.svg&quot; alt=&quot;Commit and evidence timeline&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The sequence is interesting because implementation, tests, and documentation are interleaved rather than treated as separate afterthoughts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;8c64c0c&lt;/code&gt; hardened OSPF authentication writer fallback behavior and added regression coverage.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;3d6dc52&lt;/code&gt; added an MPLS VPN data-plane regression for transport-label resolution upgrades.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;f4ed696&lt;/code&gt; fixed VPNv4 serialization so &lt;code&gt;send-community extended&lt;/code&gt; survives without forcing AF activation.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;6b33ecd&lt;/code&gt; added VRF forwarding route-ownership transition coverage.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ede88bc&lt;/code&gt; / &lt;code&gt;38f5bb1&lt;/code&gt; aligned LDP discovery CLI behavior with runtime loopback suppression and added a regression.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;894dccb&lt;/code&gt; fixed RD-safe VPN import cache deletion and reconnect-time &lt;code&gt;update-source &amp;lt;ifname&amp;gt;&lt;/code&gt; resolution, while cleaning up AF-scoped parser behavior.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;225ca52&lt;/code&gt; refreshed the implementation and project status documentation.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;f04ef1b&lt;/code&gt;, &lt;code&gt;399d04a&lt;/code&gt;, and &lt;code&gt;3a3fca4&lt;/code&gt; added parser/CLI regressions around VPNv4 shorthand and mixed-order VRF AF mutation.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;b15e645&lt;/code&gt; and &lt;code&gt;5493e3d&lt;/code&gt; consolidated the checkpoint/evidence documentation and removed duplicated readiness text.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The phase handoff recorded a full regression result of &lt;strong&gt;152/152 passing tests&lt;/strong&gt; at that checkpoint. The current repository README reports a different current test inventory, so I treat the numbers in the reports as historical evidence tied to that phase, not as an eternal project metric.&lt;/p&gt;
&lt;p&gt;This is exactly the behavior I want from AI-assisted engineering: a code change should leave behind enough evidence that a later session — human or AI — can reconstruct why it exists.&lt;/p&gt;
&lt;h2 id=&quot;the-runtime-architecture-the-agent-had-to-preserve&quot;&gt;The runtime architecture the agent had to preserve&lt;/h2&gt;
&lt;p&gt;The project is organized into focused C modules for infrastructure, platform integration, encoding, RIB, policy, routing, protocol state machines, OSPF, MPLS, configuration, and CLI.&lt;/p&gt;
&lt;p&gt;A simplified data path looks like this:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/bgpd-architecture.svg&quot; alt=&quot;bgpd-simple architecture&quot; /&gt;&lt;/p&gt;
&lt;p&gt;That architecture is another form of specification. If a task says “fix parsing under &lt;code&gt;address-family vpnv4&lt;/code&gt;,” it should not casually mutate best-path or FIB code. If an LDP CLI discrepancy is really a presentation/runtime parity issue, the plan should say exactly where the source of truth lives.&lt;/p&gt;
&lt;p&gt;A good agent plan therefore contains not only &lt;em&gt;what&lt;/em&gt; to change, but also &lt;em&gt;what must not move&lt;/em&gt;.&lt;/p&gt;
&lt;h2 id=&quot;running-qwen-3-6-27b-locally-with-llama-cpp&quot;&gt;Running Qwen 3.6 27B locally with llama.cpp&lt;/h2&gt;
&lt;p&gt;For these experiments, model inference ran locally. That was useful for two reasons: repository context stayed under my control, and I could iterate without tying the workflow to a hosted model provider.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;llama.cpp&lt;/code&gt; provides an OpenAI-compatible HTTP server, which makes it a convenient boundary between a coding agent and a local GGUF model.&lt;/p&gt;
&lt;p&gt;A typical text-only coding setup looks like:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;llama-server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /models/Qwen3.6-27B.gguf&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-alias&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; qwen3.6-27b&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-host&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 127.0.0.1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8080&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-ctx-size&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 65536&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-n-gpu-layers&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 99&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The exact GGUF quantization, context size, and GPU offload depend on the machine. I prefer to set an explicit &lt;code&gt;--alias&lt;/code&gt;; otherwise the model ID exposed by &lt;code&gt;/v1/models&lt;/code&gt; may be based on the model path, which is awkward to reference from OpenCode.&lt;/p&gt;
&lt;p&gt;Before involving the coding agent, I verify the serving boundary directly:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;curl&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; http://127.0.0.1:8080/v1/models&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and, if needed, a minimal chat request:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;curl&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; http://127.0.0.1:8080/v1/chat/completions&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;H&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;#39;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Content-Type: application/json&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;d&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;#39;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    &amp;quot;model&amp;quot;: &amp;quot;qwen3.6-27b&amp;quot;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    &amp;quot;messages&amp;quot;: [&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;      {&amp;quot;role&amp;quot;: &amp;quot;user&amp;quot;, &amp;quot;content&amp;quot;: &amp;quot;Return only: ready&amp;quot;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;    ]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;  }&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This isolates model-serving problems from agent problems.&lt;/p&gt;
&lt;h2 id=&quot;connecting-opencode-directly-to-llama-cpp&quot;&gt;Connecting OpenCode directly to llama.cpp&lt;/h2&gt;
&lt;p&gt;OpenCode supports OpenAI-compatible custom providers, and its current provider documentation includes &lt;code&gt;llama.cpp&lt;/code&gt; explicitly.&lt;/p&gt;
&lt;p&gt;A minimal project &lt;code&gt;opencode.json&lt;/code&gt; can look like this:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;json&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;$schema&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;https://opencode.ai/config.json&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;llama.cpp/qwen3.6-27b&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;provider&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;    &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;llama.cpp&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;npm&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;@ai-sdk/openai-compatible&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;llama-server (local)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;options&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;        &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;baseURL&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;http://127.0.0.1:8080/v1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      }&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;models&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;        &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;qwen3.6-27b&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;          &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Qwen 3.6 27B (local)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;          &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;limit&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;            &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;context&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 65536&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;            &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;output&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 16384&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;          }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then I start OpenCode from the repository root:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;opencode&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The basic local connection is simple:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/local-connection.svg&quot; alt=&quot;Local OpenCode to llama.cpp connection&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The coding agent sees an OpenAI-compatible model provider. It does not need to know how Qwen is loaded, quantized, or placed across GPUs.&lt;/p&gt;
&lt;h2 id=&quot;alternative-opencode-with-lm-studio&quot;&gt;Alternative: OpenCode with LM Studio&lt;/h2&gt;
&lt;p&gt;LM Studio can replace &lt;code&gt;llama-server&lt;/code&gt; without changing the Spec Kit specification, task list, or acceptance criteria. Load a model in LM Studio and enable its local server (default &lt;code&gt;127.0.0.1:1234&lt;/code&gt;). You can start the server with:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;lms&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; start&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 1234&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;curl&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; http://127.0.0.1:1234/v1/models&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The model ID must match the ID returned by your server. For example, add an LM Studio provider to &lt;code&gt;opencode.json&lt;/code&gt; and select it instead of the &lt;code&gt;llama.cpp&lt;/code&gt; provider:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;json&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;$schema&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;https://opencode.ai/config.json&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;lmstudio/my-local-model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;provider&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;    &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;lmstudio&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;npm&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;@ai-sdk/openai-compatible&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;LM Studio (local)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;options&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;baseURL&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;http://127.0.0.1:1234/v1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;      &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;models&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;        &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;my-local-model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Local model (replace ID)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;my-local-model&lt;/code&gt; is a placeholder; replace it in both locations with a real ID from &lt;code&gt;/v1/models&lt;/code&gt;. This JSON illustrates an alternative configuration, not a second &lt;code&gt;opencode.json&lt;/code&gt; to place alongside the original. LM Studio and llama.cpp expose similar OpenAI-style interfaces, but sampling, tool-call parsing, and context handling can differ by runtime and model.&lt;/p&gt;
&lt;h2 id=&quot;alternative-coding-agent-claude-code&quot;&gt;Alternative coding agent: Claude Code&lt;/h2&gt;
&lt;p&gt;Spec Kit also supports &lt;strong&gt;Claude Code&lt;/strong&gt; directly. In current Spec Kit versions, a fresh project can be initialized with &lt;code&gt;--integration claude&lt;/code&gt; rather than &lt;code&gt;--integration opencode&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;specify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; init&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-here&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-force&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-integration&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; claude&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;claude&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For an &lt;strong&gt;existing Spec Kit project&lt;/strong&gt;, avoid reinitializing unnecessarily. To install Claude alongside OpenCode (in Spec Kit versions supporting multiple integrations), use:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;specify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; integration&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; install&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; claude&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If your installed version reports an integration conflict, inspect &lt;code&gt;specify integration list&lt;/code&gt; and the relevant &lt;code&gt;switch&lt;/code&gt;/&lt;code&gt;--force&lt;/code&gt; guidance before changing existing integration files. Claude Code&#39;s Spec Kit skills live under &lt;code&gt;.claude/skills/&lt;/code&gt;, while OpenCode uses its own integration directory. The shared &lt;code&gt;.specify/&lt;/code&gt; artifacts remain the workflow&#39;s common source of truth.&lt;/p&gt;
&lt;p&gt;Claude Code normally uses hosted Anthropic models. It can &lt;strong&gt;also act as a local-model client&lt;/strong&gt; when the inference server supports the Anthropic Messages API. That does &lt;em&gt;not&lt;/em&gt; turn Qwen into an Anthropic Claude model: Claude Code is the CLI agent, and the locally served model is still Qwen (or another loaded model).&lt;/p&gt;
&lt;h3 id=&quot;claude-code-lm-studio&quot;&gt;Claude Code + LM Studio&lt;/h3&gt;
&lt;p&gt;LM Studio exposes &lt;code&gt;POST /v1/messages&lt;/code&gt; as an Anthropic-compatible endpoint. With the model already loaded:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;lms&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; start&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 1234&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;export&lt;/span&gt;&lt;span&gt; ANTHROPIC_BASE_URL&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span&gt;http&lt;/span&gt;&lt;span&gt;://&lt;/span&gt;&lt;span&gt;127&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;0&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;0&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;1&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt;1234&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;export&lt;/span&gt;&lt;span&gt; ANTHROPIC_AUTH_TOKEN&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span&gt;lmstudio&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;claude&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; YOUR_LM_STUDIO_MODEL_ID&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Use the actual model identifier from LM Studio. The placeholder token is sufficient only when server authentication is disabled; if enabled, use a real LM Studio API token. Keep these environment variables scoped to the terminal/session using the local server so that subsequent Claude Code sessions are not accidentally redirected.&lt;/p&gt;
&lt;h3 id=&quot;claude-code-llama-cpp&quot;&gt;Claude Code + llama.cpp&lt;/h3&gt;
&lt;p&gt;Recent &lt;code&gt;llama-server&lt;/code&gt; builds also expose Anthropic-compatible &lt;code&gt;/v1/messages&lt;/code&gt;. Start a tool-capable model with a compatible chat template and enable &lt;code&gt;--jinja&lt;/code&gt; for tool calls:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;llama-server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /models/YOUR-MODEL.gguf&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-alias&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; local-coder&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-host&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 127.0.0.1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8080&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-ctx-size&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 65536&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-jinja&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;export&lt;/span&gt;&lt;span&gt; ANTHROPIC_BASE_URL&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span&gt;http&lt;/span&gt;&lt;span&gt;://&lt;/span&gt;&lt;span&gt;127&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;0&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;0&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;1&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;span&gt;8080&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;export&lt;/span&gt;&lt;span&gt; ANTHROPIC_AUTH_TOKEN&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span&gt;local&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;claude&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-model&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; local-coder&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;ANTHROPIC_BASE_URL&lt;/code&gt; deliberately has &lt;strong&gt;no &lt;code&gt;/v1&lt;/code&gt; suffix&lt;/strong&gt; in these Claude Code commands; the client appends the Messages API route. As with LM Studio, tool-use behavior and streaming compatibility depend on the server build, template, context size, and model. Verify a small edit plus tool execution before starting a large RFC-driven task.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/agent-inference-matrix.svg&quot; alt=&quot;Agent and server connection matrix&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The practical combinations are:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Coding agent&lt;/th&gt;&lt;th&gt;llama.cpp&lt;/th&gt;&lt;th&gt;LM Studio&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;OpenCode&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;OpenAI-compatible &lt;code&gt;:8080/v1&lt;/code&gt;&lt;/td&gt;&lt;td&gt;OpenAI-compatible &lt;code&gt;:1234/v1&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Claude Code&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;Anthropic-compatible &lt;code&gt;:8080/v1/messages&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Anthropic-compatible &lt;code&gt;:1234/v1/messages&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;In this architecture, &lt;strong&gt;the Spec Kit plan and the repository tests stay the same even if the agent CLI or serving engine changes&lt;/strong&gt;. Switching the coding agent may still require its matching Spec Kit integration files; switching the inference server does not.&lt;/p&gt;
&lt;h2 id=&quot;other-connection-patterns&quot;&gt;Other connection patterns&lt;/h2&gt;
&lt;h3 id=&quot;1-llama-cpp-on-another-workstation&quot;&gt;1. llama.cpp on another workstation&lt;/h3&gt;
&lt;p&gt;If the GPU machine is on a trusted private network, &lt;code&gt;llama-server&lt;/code&gt; can listen on a non-loopback address:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;llama-server&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /models/Qwen3.6-27B.gguf&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-alias&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; qwen3.6-27b&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-host&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0.0.0.0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8080&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and OpenCode can point to:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;json&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;baseURL&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;http://gpu-workstation:8080/v1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I would not expose this unauthenticated endpoint directly to the public Internet.&lt;/p&gt;
&lt;h3 id=&quot;2-remote-gpu-machine-through-an-ssh-tunnel&quot;&gt;2. Remote GPU machine through an SSH tunnel&lt;/h3&gt;
&lt;p&gt;For a machine outside the local trusted network, an SSH tunnel keeps the OpenCode configuration unchanged:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ssh&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;L&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 8080:127.0.0.1:8080&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; user@gpu-host&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;llama-server&lt;/code&gt; still binds to &lt;code&gt;127.0.0.1&lt;/code&gt; on the remote machine, while OpenCode still connects to &lt;code&gt;http://127.0.0.1:8080/v1&lt;/code&gt; locally.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/spec-driven-ai-development-bgpd-simple/remote-connection.svg&quot; alt=&quot;Remote model through SSH tunnel&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This is my preferred remote pattern because the inference endpoint itself remains loopback-only.&lt;/p&gt;
&lt;h3 id=&quot;3-llama-cpp-in-docker&quot;&gt;3. llama.cpp in Docker&lt;/h3&gt;
&lt;p&gt;The same boundary also works when the model server is containerized:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;docker&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; run&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-rm&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-gpus&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; all&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;p&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 127.0.0.1:8080:8080&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;v&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /models:/models&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;  ghcr.io/ggml-org/llama.cpp:server-cuda&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;m&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /models/Qwen3.6-27B.gguf&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-alias&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; qwen3.6-27b&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-host&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0.0.0.0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-port&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8080&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; \&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;  -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-n-gpu-layers&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 99&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;OpenCode does not change. That is another advantage of keeping an HTTP compatibility boundary between the agent and inference runtime.&lt;/p&gt;
&lt;h2 id=&quot;how-i-use-the-agent-during-implementation&quot;&gt;How I use the agent during implementation&lt;/h2&gt;
&lt;p&gt;A local model is still an agentic coding model, not an oracle. I get better results when the work is segmented around evidence.&lt;/p&gt;
&lt;p&gt;For a protocol change I normally ask OpenCode to work through a bounded task such as:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Implement task T04 only.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Read the feature spec and plan first.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Use RFC 4271 and the project&amp;#39;s BGP compliance document as authority.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Do not change unrelated CLI syntax.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Add the focused regression before considering the task complete.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Run the focused test, then the full relevant BGP gate.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Summarize changed files and verification evidence.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The useful part is that most of those instructions should already exist in project artifacts. The prompt becomes a selector and an execution boundary, not the sole memory of the project.&lt;/p&gt;
&lt;p&gt;If the model starts making broad changes, that is usually a signal that the plan is too vague. If it repeatedly fails the same test, the project&#39;s workflow rules say to stop and re-plan rather than continue patching symptoms. This prevents the “agent keeps editing until the test happens to pass” failure mode.&lt;/p&gt;
&lt;h2 id=&quot;verification-is-part-of-implementation&quot;&gt;Verification is part of implementation&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;bgpd-simple&lt;/code&gt; has dedicated compliance and soak tooling in addition to normal unit/regression tests. The project reports reference commands such as:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ctest&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-test-dir&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; build&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-output-on-failure&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and targeted protocol slices, plus scripts such as:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;tools/run_bgp_mpls_compliance_gate.sh&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;tools/protocol_soak_evidence.sh&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;tools/mpls_ldp_soak_gate.sh&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is where spec-driven AI development becomes materially different from “AI generated the patch.”&lt;/p&gt;
&lt;p&gt;The implementation is not done when the agent says it is done. It is done when:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;the requested behavior is present;&lt;/li&gt;
&lt;li&gt;the focused regression passes;&lt;/li&gt;
&lt;li&gt;the relevant protocol gate passes;&lt;/li&gt;
&lt;li&gt;unrelated regressions do not appear;&lt;/li&gt;
&lt;li&gt;compliance documentation still matches reality;&lt;/li&gt;
&lt;li&gt;the commit explains the behavioral boundary.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The repository&#39;s release rules capture the same idea: no in-scope &lt;code&gt;FAIL&lt;/code&gt; items, explicit constraints for partial support, and compliance claims backed by both implementation references and executable evidence.&lt;/p&gt;
&lt;h2 id=&quot;where-local-models-work-well-and-where-they-do-not&quot;&gt;Where local models work well — and where they do not&lt;/h2&gt;
&lt;p&gt;In my use, Qwen 3.6 27B was useful for repository-level work, especially when the task had a narrow spec and strong tests. But a local model benefits even more than a frontier hosted model from reducing ambiguity.&lt;/p&gt;
&lt;p&gt;The highest-value uses were:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;tracing behavior across parser, protocol, RIB, CLI, and tests;&lt;/li&gt;
&lt;li&gt;generating focused regression tests from a concrete failure;&lt;/li&gt;
&lt;li&gt;implementing bounded changes after the plan was accepted;&lt;/li&gt;
&lt;li&gt;keeping documentation and test evidence synchronized;&lt;/li&gt;
&lt;li&gt;checking repetitive RFC-related conditions across multiple code paths.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The weaker pattern is giving the model a broad request such as “make OSPF fully compliant” and hoping it will discover scope, architecture, protocol semantics, test strategy, and release criteria by itself.&lt;/p&gt;
&lt;p&gt;That is not a model-size problem. It is a requirements problem.&lt;/p&gt;
&lt;h2 id=&quot;what-spec-driven-development-changed-for-me&quot;&gt;What Spec-Driven Development changed for me&lt;/h2&gt;
&lt;p&gt;The biggest change was not code generation speed. It was &lt;strong&gt;context durability&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;A chat session is temporary. A specification, compliance matrix, regression test, and commit are durable. When the next task begins, OpenCode and the local model can re-read the same artifacts that a human maintainer would use.&lt;/p&gt;
&lt;p&gt;The workflow also makes model replacement much less disruptive. I can use Qwen locally today, another local model tomorrow, or a hosted model for a difficult task. The process does not disappear with the model because the important state lives in the repository.&lt;/p&gt;
&lt;p&gt;For protocol software, that is the direction I want AI development to move:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;requirements as versioned artifacts, implementation as a constrained transformation, and verification as executable evidence.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The model is a powerful participant in that loop, but it is not the loop itself.&lt;/p&gt;
&lt;h2 id=&quot;references&quot;&gt;References&lt;/h2&gt;
&lt;p&gt;Project:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple&quot;&gt;&lt;code&gt;avkghost/bgpd-simple&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple/blob/main/docs/meta/WO.md&quot;&gt;&lt;code&gt;docs/meta/WO.md&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple/blob/main/docs/reports/IMPLEMENTATION_STATUS.md&quot;&gt;&lt;code&gt;docs/reports/IMPLEMENTATION_STATUS.md&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple/blob/main/docs/reports/PHASE_HANDOFF_2026-03-07.md&quot;&gt;&lt;code&gt;docs/reports/PHASE_HANDOFF_2026-03-07.md&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/bgpd-simple/blob/main/docs/reports/RFC_COMPLIANCE_SUMMARY.md&quot;&gt;&lt;code&gt;docs/reports/RFC_COMPLIANCE_SUMMARY.md&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Tooling:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/github/spec-kit&quot;&gt;GitHub Spec Kit&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/github/spec-kit/blob/main/docs/integrations/index.md&quot;&gt;Spec Kit: supported integrations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/github/spec-kit/blob/main/docs/how-to/existing-projects.md&quot;&gt;Spec Kit: adopting an existing project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://opencode.ai/docs/providers/&quot;&gt;OpenCode providers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://dev.opencode.ai/docs/providers&quot;&gt;LM Studio: OpenCode-compatible provider configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://lmstudio.ai/docs/integrations/claude-code&quot;&gt;LM Studio: Claude Code integration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://lmstudio.ai/docs/developer/anthropic-compat&quot;&gt;LM Studio: Anthropic-compatible API&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/github/spec-kit/blob/main/docs/integrations/index.md&quot;&gt;Spec Kit: managing multiple agent integrations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md&quot;&gt;llama.cpp server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://huggingface.co/Qwen/Qwen3.6-27B&quot;&gt;Qwen3.6-27B model card&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Libella — Privacy Policy</title>
        <published>2026-10-14T00:00:00+00:00</published>
        <updated>2026-10-14T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/apps/libella/privacy/"/>
        <id>https://iamavk.info/apps/libella/privacy/</id>
        
        <content type="html" xml:base="https://iamavk.info/apps/libella/privacy/">&lt;p&gt;&lt;strong&gt;Effective date:&lt;/strong&gt; 14 October 2026&lt;/p&gt;
&lt;p&gt;This policy explains how Libella handles information. Libella is provided by &lt;strong&gt;ANDREI KAZIALETSKI, trading as AVKSoft&lt;/strong&gt;.
Privacy questions can be sent to &lt;a href=&quot;mailto:libella@iamavk.info&quot;&gt;libella@iamavk.info&lt;/a&gt;. No postal address is required to contact us.&lt;/p&gt;
&lt;h2 id=&quot;documents-and-local-data&quot;&gt;Documents and local data&lt;/h2&gt;
&lt;p&gt;Libella is designed as an offline PDF/DjVu reader. Imported documents are copied into app-managed local storage and
processed locally. Documents, library metadata, bookmarks and reading positions are not uploaded to AVKSoft servers or
AVKSoft cloud services. Libella has no AVKSoft account system and no proprietary cloud synchronization or backend.&lt;/p&gt;
&lt;p&gt;Apple or Google operating-system or cloud backups may include application data according to the user&#39;s platform and
account settings. Those backups are outside AVKSoft&#39;s direct control.&lt;/p&gt;
&lt;h2 id=&quot;analytics-advertising-and-tracking&quot;&gt;Analytics, advertising and tracking&lt;/h2&gt;
&lt;p&gt;For Libella specifically, there are no ads and no AVKSoft analytics or tracking used to profile reading behavior.
Document and reading activity is not used for advertising. AVKSoft does not sell personal data.&lt;/p&gt;
&lt;h2 id=&quot;store-purchases&quot;&gt;Store purchases&lt;/h2&gt;
&lt;p&gt;Libella may communicate with the Apple App Store or Google Play for product information, localized pricing, purchase
initiation, purchase restoration or recovery, and determining whether a Pro entitlement is active. Apple and Google
process payments under their own terms and privacy policies. AVKSoft does not receive or store payment card details.
Apple App Store and Google Play purchases are separate, and Libella has no account or backend that transfers purchases
between stores. Store availability and recovery behavior should be confirmed with the relevant store setup.&lt;/p&gt;
&lt;h2 id=&quot;support-communications&quot;&gt;Support communications&lt;/h2&gt;
&lt;p&gt;The app does not require personal contact details. If you voluntarily email support, AVKSoft may receive your email
address, message contents, device/OS/app-version information, screenshots, and files or documents you attach. This
information is used to respond to support requests and diagnose reported problems, not for advertising. Please avoid
sending confidential or sensitive documents unless necessary. Support correspondence is kept for as long as reasonably
necessary to handle the request, maintain relevant support records, or comply with legal obligations. You may request
deletion at &lt;a href=&quot;mailto:libella@iamavk.info&quot;&gt;libella@iamavk.info&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;sharing&quot;&gt;Sharing&lt;/h2&gt;
&lt;p&gt;AVKSoft does not sell personal data or share documents or reading activity with advertisers or data brokers. Apple and
Google may process information in connection with their store, operating-system, payment and backup services. Email
infrastructure providers necessarily process support correspondence. Information may also be disclosed when required by law.&lt;/p&gt;
&lt;h2 id=&quot;retention-and-deletion&quot;&gt;Retention and deletion&lt;/h2&gt;
&lt;p&gt;Local app-managed data remains on the device or in app storage. Deleting a document removes the app-managed copy
according to the app&#39;s behavior. Uninstalling normally removes Libella&#39;s local app data, but original files may remain
at their source location and operating-system or cloud backups may have their own retention.&lt;/p&gt;
&lt;h2 id=&quot;privacy-rights&quot;&gt;Privacy rights&lt;/h2&gt;
&lt;p&gt;Depending on applicable law, you may request access, correction, deletion, or restriction or objection where applicable,
and may complain to a supervisory authority. In Poland, this includes the &lt;strong&gt;Prezes Urzędu Ochrony Danych Osobowych
(UODO)&lt;/strong&gt;. Contact &lt;a href=&quot;mailto:libella@iamavk.info&quot;&gt;libella@iamavk.info&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;children-s-privacy&quot;&gt;Children&#39;s privacy&lt;/h2&gt;
&lt;p&gt;Libella has no account, advertising, or behavioral analytics/profile system. This policy makes no unsupported age-certification claim.&lt;/p&gt;
&lt;h2 id=&quot;security&quot;&gt;Security&lt;/h2&gt;
&lt;p&gt;AVKSoft uses reasonable measures appropriate to the information handled, but no method of storage or transmission can
be guaranteed completely secure.&lt;/p&gt;
&lt;h2 id=&quot;changes-to-this-policy&quot;&gt;Changes to this policy&lt;/h2&gt;
&lt;p&gt;If this policy changes, the new version will be published on this page with a new effective date. For questions, contact
&lt;a href=&quot;mailto:libella@iamavk.info&quot;&gt;libella@iamavk.info&lt;/a&gt; or see the &lt;a href=&quot;https://iamavk.info/apps/libella/support/&quot;&gt;Libella Support page&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Chasing a PCIe Ghost on Raspberry Pi 5</title>
        <published>2026-10-14T00:00:00+00:00</published>
        <updated>2026-10-14T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/raspberry-pi-5-pcie-debugging-story/"/>
        <id>https://iamavk.info/blog/raspberry-pi-5-pcie-debugging-story/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/raspberry-pi-5-pcie-debugging-story/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/raspberry-pi-5-pcie-debugging-story/hero.png&quot; alt=&quot;Abstract illustration of Raspberry Pi PCIe link-training debugging&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;How a simple expansion experiment turned into a deep dive through link training, oscilloscopes, kernel patches, and a plan to build my own PCB.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-setup&quot;&gt;The setup&lt;/h2&gt;
&lt;p&gt;I wanted to connect standard PCIe cards to a Raspberry Pi 5. I used the &lt;strong&gt;52Pi P02 (EP-0219)&lt;/strong&gt; adapter: a full-size PCIe slot connected to the Pi&#39;s external PCIe FFC connector. The Pi exposes one lane, so even physically wider cards operate at x1.&lt;/p&gt;
&lt;p&gt;My test hardware included an &lt;strong&gt;AXAGON PCEE-G25 RTL8125B&lt;/strong&gt; 2.5GbE NIC, a &lt;strong&gt;10Gtek four-port RTL8125&lt;/strong&gt; NIC behind an &lt;strong&gt;ASMedia ASM1806&lt;/strong&gt; switch, a standalone &lt;strong&gt;ASM1184E&lt;/strong&gt; switch card, and a &lt;strong&gt;Broadcom/LSI SAS9300-8i (SAS3008)&lt;/strong&gt; HBA.&lt;/p&gt;
&lt;p&gt;The results were not what I expected.&lt;/p&gt;
&lt;h2 id=&quot;a-puzzling-compatibility-matrix&quot;&gt;A puzzling compatibility matrix&lt;/h2&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Device&lt;/th&gt;&lt;th&gt;Raspberry Pi 5 + P02 result&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;AXAGON RTL8125B&lt;/td&gt;&lt;td&gt;No link; ends in &lt;code&gt;Polling.Compliance&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;ASM1184E&lt;/td&gt;&lt;td&gt;No usable link; sometimes reaches &lt;code&gt;Polling.Configuration&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;10Gtek ASM1806&lt;/td&gt;&lt;td&gt;Upstream link works; switch and downstream bridges enumerate&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;10Gtek 4× RTL8125&lt;/td&gt;&lt;td&gt;No NIC endpoints enumerate&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;SAS3008&lt;/td&gt;&lt;td&gt;Works, including &lt;strong&gt;Gen3 8.0 GT/s x1&lt;/strong&gt;, with no reported AER errors&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;The SAS3008 was a particularly useful control. It demonstrated that the same Pi, FFC cable, and P02 could carry a working Gen3 link. But that did not prove every electrical and timing characteristic of the adapter was correct for every endpoint.&lt;/p&gt;
&lt;h2 id=&quot;watching-pcie-link-training&quot;&gt;Watching PCIe link training&lt;/h2&gt;
&lt;p&gt;I enabled the Raspberry Pi Broadcom driver&#39;s LTSSM tracing:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;pcie-brcmstb.trace_ltssm=1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For the AXAGON, the result was repeatable across Gen1, Gen2, and Gen3 settings:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Detect.Active&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Polling.Active&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;~24 ms later: Polling.Compliance&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;link down&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The ASM1184E behaved differently. At Gen2/Gen3 it could reach &lt;code&gt;Polling.Configuration&lt;/code&gt;, but still never established a usable link. Working cards advanced through &lt;code&gt;Configuration.Link&lt;/code&gt;, the other configuration substates, and ultimately &lt;code&gt;L0&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;These are different failure stages, not one universal symptom.&lt;/p&gt;
&lt;h2 id=&quot;ruling-out-the-obvious&quot;&gt;Ruling out the obvious&lt;/h2&gt;
&lt;p&gt;I tested Raspberry Pi kernel &lt;strong&gt;6.12&lt;/strong&gt; and &lt;strong&gt;6.18&lt;/strong&gt;; the behavior persisted. I forced Gen1/Gen2/Gen3, tried a &lt;strong&gt;500 ms&lt;/strong&gt; delay before PERST# release, changed the P02&#39;s power arrangement, and tested with CLKREQ held low. I checked the 5 V supply and PCIe slot&#39;s 3.3 V rails. None of these changes fixed the failing cards.&lt;/p&gt;
&lt;p&gt;With a &lt;strong&gt;FNIRSI DPOX180H&lt;/strong&gt; oscilloscope, I verified that PERST# was released during the link-training window and reasserted after failure. I also observed an approximately &lt;strong&gt;100 MHz REFCLK&lt;/strong&gt; during training on the AXAGON and ASM1184E. So the reset and clock signals were not simply absent.&lt;/p&gt;
&lt;p&gt;The scope cannot resolve a PCIe 2.5 GT/s eye diagram, but it can reveal coarse activity. The AXAGON showed activity from the card toward the Pi; comparable Pi-to-card training activity was not clearly visible. The ASM1184E showed Pi-to-card activity and progressed farther in the LTSSM.&lt;/p&gt;
&lt;p&gt;An approximately &lt;strong&gt;208 mV&lt;/strong&gt; receiver-detect-looking event was visible with an empty slot, and a similar event appeared with the working 10Gtek. I did not see an equally clear event with the AXAGON. This was an interesting comparison, &lt;strong&gt;not a compliance-grade measurement or proof of a particular receiver-detection failure&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Eventually, probing the tiny FFC connector with ordinary multimeter probes became more risky than useful. Adjacent contacts were too easy to bridge, and continuity readings could be misleading.&lt;/p&gt;
&lt;h2 id=&quot;a-kernel-patch-retry-without-perst&quot;&gt;A kernel patch: retry without PERST#&lt;/h2&gt;
&lt;p&gt;One hypothesis was that the RTL8125 needed more time after startup. I patched the &lt;strong&gt;Raspberry Pi 6.18 &lt;code&gt;pcie-brcmstb&lt;/code&gt; driver&lt;/strong&gt; to toggle the root port&#39;s &lt;strong&gt;Link Disable&lt;/strong&gt; bit after the initial failure, restarting link training &lt;strong&gt;without reasserting PERST#&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The first attempt produced:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Detect.Active → Polling.Active → Polling.Compliance&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The retry produced the &lt;strong&gt;same sequence&lt;/strong&gt;. The second capture reported zero timestamps because tracing began after the restart, so those timestamps were not useful for timing analysis. The meaningful result was that &lt;strong&gt;the warm LTSSM retry did not recover the AXAGON&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;This weakened the simple “endpoint was not ready yet” explanation, although it did not reveal the underlying cause.&lt;/p&gt;
&lt;h2 id=&quot;the-x86-cross-check-changed-everything&quot;&gt;The x86 cross-check changed everything&lt;/h2&gt;
&lt;p&gt;Later I tested the cards in an x86-64 PC. &lt;strong&gt;Every card worked.&lt;/strong&gt; The ASM1184E exposed its four ports; the 10Gtek enumerated its &lt;strong&gt;ASM1806 switch and all four RTL8125 NICs&lt;/strong&gt;; and the AXAGON worked. A separate PCIe x1-to-x16 riser/extender also successfully detected an RTL8127.&lt;/p&gt;
&lt;p&gt;This ruled out treating the endpoint cards as generally defective.&lt;/p&gt;
&lt;p&gt;It also exposed a problem with the early “RTL8125 versus BCM2712” theory: on the 10Gtek, the four RTL8125 controllers are &lt;strong&gt;behind the ASM1806 switch&lt;/strong&gt;, not directly connected to the Raspberry Pi PHY. A direct BCM2712-to-RTL8125 electrical mismatch alone cannot explain every observation.&lt;/p&gt;
&lt;p&gt;The common failure environment is the &lt;strong&gt;Pi 5 + FFC cable + 52Pi P02 combination&lt;/strong&gt;. Plausible causes still include P02 signal integrity or power/reset implementation, BCM2712 external PCIe interoperability, or an interaction between them. Successful HATs such as Hailo setups show that the Pi&#39;s external PCIe can work, but they do not isolate the P02 because they replace the adapter path entirely.&lt;/p&gt;
&lt;h2 id=&quot;the-adapter-became-the-unknown&quot;&gt;The adapter became the unknown&lt;/h2&gt;
&lt;p&gt;I consulted the &lt;a rel=&quot;external&quot; href=&quot;https://wiki.52pi.com/index.php?title=EP-0219&quot;&gt;52Pi EP-0219 wiki&lt;/a&gt; and the &lt;a rel=&quot;external&quot; href=&quot;https://pip-assets.raspberrypi.com/categories/892-raspberry-pi-5/documents/RP-008298-DS-1-pcie-connector-standard.pdf&quot;&gt;official Raspberry Pi 5 PCIe connector specification&lt;/a&gt;. The official J20 signal assignment includes:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th style=&quot;text-align: right&quot;&gt;J20 pin&lt;/th&gt;&lt;th&gt;Signal&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;4 / 5&lt;/td&gt;&lt;td&gt;PCIe reference clock P / N&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;7 / 8&lt;/td&gt;&lt;td&gt;PCIe RX P / N&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;10 / 11&lt;/td&gt;&lt;td&gt;PCIe TX P / N&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;13&lt;/td&gt;&lt;td&gt;PCIe power enable&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;14&lt;/td&gt;&lt;td&gt;PCIe detect / wake&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;15&lt;/td&gt;&lt;td&gt;PCIe CLKREQ#&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td style=&quot;text-align: right&quot;&gt;16&lt;/td&gt;&lt;td&gt;PCIe reset&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Without a full, verified P02 schematic and layout—or another equivalent PCIe slot adapter for A/B testing—I could not confidently separate the board&#39;s behavior from the Pi&#39;s own PCIe behavior. Basic continuity measurements were not going to tell me enough about AC coupling, controlled impedance, clock routing, and reset or power sequencing.&lt;/p&gt;
&lt;h2 id=&quot;next-designing-my-own-pcie-pcb&quot;&gt;Next: designing my own PCIe PCB&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;I&#39;m going to design and build a custom PCB with all required onboard power converters and a full-size PCIe connector, replacing the 52Pi P02 adapter in my test setup.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The design will include a Pi 5-compatible FFC interface, regulated &lt;strong&gt;12 V and 3.3 V slot supplies&lt;/strong&gt; sized for the intended cards, controlled-impedance PCIe TX/RX and REFCLK routing, correct reset and sideband handling, and accessible test points for power and low-speed signals. The slot will accept wider physical cards, while the Pi connection remains &lt;strong&gt;one PCIe lane&lt;/strong&gt;. I&#39;ll also make the schematic, component choices, and power sequencing fully inspectable.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;I have not proven that the P02 is defective.&lt;/strong&gt; The purpose of the custom PCB is to remove an opaque adapter from the experiment and make the entire signal and power path understandable and measurable.&lt;/p&gt;
&lt;p&gt;If the same cards fail on the new PCB, the evidence will shift toward Pi 5 / BCM2712 behavior or a shared design assumption. If they work, the P02 implementation or its connection will become the much stronger suspect.&lt;/p&gt;
&lt;p&gt;Either outcome is progress. The next chapter will be about designing, assembling, and testing the replacement board.&lt;/p&gt;
&lt;hr /&gt;
&lt;h3 id=&quot;references&quot;&gt;References&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://wiki.52pi.com/index.php?title=EP-0219&quot;&gt;52Pi P02 / EP-0219 wiki&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://pip-assets.raspberrypi.com/categories/892-raspberry-pi-5/documents/RP-008298-DS-1-pcie-connector-standard.pdf&quot;&gt;Raspberry Pi 5 PCIe connector standard (PDF)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/raspberrypi/linux/tree/rpi-6.18.y&quot;&gt;Raspberry Pi Linux kernel &lt;code&gt;rpi-6.18.y&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>From BlueSCSI Ultra to DaynaPORT: Connecting Legacy SCSI Devices to Linux and Windows</title>
        <published>2026-08-16T00:00:00+00:00</published>
        <updated>2026-08-16T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/bluescsi-daynaport-linux-windows/"/>
        <id>https://iamavk.info/blog/bluescsi-daynaport-linux-windows/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/bluescsi-daynaport-linux-windows/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/hero.png&quot; alt=&quot;Abstract illustration of legacy SCSI hardware connected to modern systems&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The project did not start with networking. It started with a much more practical problem: &lt;strong&gt;I wanted to use my old SCSI devices again.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;I had devices such as a &lt;strong&gt;Sony SMO-F551&lt;/strong&gt; magneto-optical drive and &lt;strong&gt;Iomega Jaz 1 GB / 2 GB&lt;/strong&gt; drives. The drives were still useful to me, but connecting them to a modern computer was the difficult part. At the time, I could not find a suitable old SCSI controller.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/sony-smo-f551-rear.jpg&quot; alt=&quot;Rear panel of the Sony SMO-F551 magneto-optical drive&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Sony SMO-F551 viewed from the rear, showing the physical SCSI connection area.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Instead, I used a &lt;strong&gt;BlueSCSI Ultra&lt;/strong&gt; and began exploring how far its bridge capabilities could be taken. My original goal was straightforward: give a modern Linux or Windows host access to physical legacy SCSI peripherals, without first having to source a period-correct SCSI host adapter.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/jaz-bluescsi-setup.jpg&quot; alt=&quot;Jaz drive and BlueSCSI Ultra in the test setup&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The test setup: BlueSCSI Ultra behind the Jaz drive, with the drive heads parked.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/jaz-drive-detail.jpg&quot; alt=&quot;Additional view of the Jaz drive in the test setup&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;A closer view of the Jaz drive and its connected cabling.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;That distinction matters. BlueSCSI is widely known for &lt;em&gt;emulating&lt;/em&gt; SCSI peripherals, such as hard disks backed by SD-card images. My problem was the other way around: I already had real SCSI devices and wanted to communicate with them.&lt;/p&gt;
&lt;h3 id=&quot;how-networking-entered-the-picture&quot;&gt;How networking entered the picture&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/legacy-scsi-bridge.svg&quot; alt=&quot;Modern computer connected to physical Sony and Iomega SCSI drives through BlueSCSI Ultra&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The original goal: access real legacy SCSI hardware without a vintage host adapter.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Working on the SCSI bridge exposed another possibility. BlueSCSI also has DaynaPORT-related functionality, based on the &lt;strong&gt;DaynaPORT SCSI/Link&lt;/strong&gt; network adapter protocol from the 1990s. It was worth exploring whether the bridge and a host-side driver could expose that interface to Linux and Windows as an ordinary network adapter.&lt;/p&gt;
&lt;p&gt;That led to the work described here: a BlueSCSI bridge modification that became
stable under Windows, a working Linux DaynaPORT driver, and a Windows
DaynaPORT driver family under development and validation. The network
interface was an extension of the original SCSI-access project, &lt;strong&gt;not the
reason I bought or started using BlueSCSI Ultra&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The work now spans three repositories:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;BlueSCSI-v2&lt;/code&gt;, with my experimental &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/BlueSCSI-v2/tree/avkghost/birdge-with-daynaport&quot;&gt;&lt;code&gt;avkghost/birdge-with-daynaport&lt;/code&gt;&lt;/a&gt; branch&lt;/li&gt;
&lt;li&gt;&lt;code&gt;daynaport-scsilink-linux-driver&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;daynaport-scsilink-windows-driver&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The interesting part is not only how the working Linux driver can expose the device as a network card. It is that the same basic DaynaPORT protocol has to survive several very different environments:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;native SCSI,&lt;/li&gt;
&lt;li&gt;BlueSCSI&#39;s USB bridge,&lt;/li&gt;
&lt;li&gt;Linux kernels from 2.0 to 7.x,&lt;/li&gt;
&lt;li&gt;Windows 9x,&lt;/li&gt;
&lt;li&gt;Windows NT 4.0,&lt;/li&gt;
&lt;li&gt;Windows 2000/XP,&lt;/li&gt;
&lt;li&gt;and Windows 10/11.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The result is a useful example of what happens when a packet protocol from the early 1990s meets several decades of operating-system driver architecture.&lt;/p&gt;
&lt;p&gt;There is an important status distinction. The modified BlueSCSI bridge has
been confirmed to operate stably under Windows, without crashes in the tested
block-device scenario. The Linux driver is working and its behavior has been
confirmed by testing. The first result confirms the device-side bridge and
Windows block-device integration; neither result confirms the separate Windows
DaynaPORT host driver. The Windows repository contains the driver
implementations and project files, but this repository does not include
reproducible Windows driver build or installation evidence. It may have been
built outside the repository, possibly with a MinGW-based DDK, but
compatibility with the corresponding Microsoft WDK and the Windows host-driver
path still needs to be confirmed. The Windows work described below is
therefore an engineering port and validation effort, not a claim of a released
or fully validated driver.&lt;/p&gt;
&lt;p&gt;The Windows repository does include host-side C89 tests for the shared protocol
core, and a GitHub Actions workflow runs those tests. They cover CDB assembly,
receive parsing, chunk reassembly, malformed records, and polling policy; they
do not compile, install, sign, or exercise a Windows kernel driver.&lt;/p&gt;
&lt;h2 id=&quot;daynaport-is-not-a-disk&quot;&gt;DaynaPORT is not a disk&lt;/h2&gt;
&lt;p&gt;A DaynaPORT SCSI/Link appears on the SCSI bus as a &lt;strong&gt;Processor device&lt;/strong&gt; (&lt;code&gt;type 0x03&lt;/code&gt;), not as a Direct Access device.&lt;/p&gt;
&lt;p&gt;The host discovers it with a normal SCSI &lt;code&gt;INQUIRY&lt;/code&gt;, looking for something equivalent to:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Vendor:  Dayna&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Product: SCSI/Link&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Type:    Processor (0x03)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;From there, Ethernet frames are moved with ordinary-looking six-byte SCSI commands that have Dayna-specific semantics.&lt;/p&gt;
&lt;p&gt;The core operations are surprisingly small:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;INQUIRY          0x12   identify the device&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;READ(6)          0x08   receive Ethernet frames&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;WRITE(6)         0x0A   transmit Ethernet frames&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;RETRIEVE STATS   0x09   retrieve MAC address and counters&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;ENABLE/DISABLE   0x0E   bring the interface up or down&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A Linux or Windows driver therefore sits between two completely different subsystems:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/network-stack.svg&quot; alt=&quot;DaynaPORT network data path&quot; /&gt;&lt;/p&gt;
&lt;p&gt;BlueSCSI itself does not run DHCP for the host and does not own the host&#39;s IP configuration. In this mode it behaves much more like an Ethernet bridge: the operating system still owns the network stack.&lt;/p&gt;
&lt;h2 id=&quot;the-first-important-limitation-there-is-no-rx-interrupt&quot;&gt;The first important limitation: there is no RX interrupt&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/daynaport-data-path.svg&quot; alt=&quot;DaynaPORT network stack and SCSI data path&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The operating system handles networking; the DaynaPORT driver converts Ethernet traffic to SCSI commands.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;A normal Ethernet controller raises an interrupt when a packet arrives.&lt;/p&gt;
&lt;p&gt;DaynaPORT does not.&lt;/p&gt;
&lt;p&gt;The only way to discover incoming traffic is to issue &lt;code&gt;READ(6)&lt;/code&gt; repeatedly.&lt;/p&gt;
&lt;p&gt;That creates one of the central design problems in the entire driver.&lt;/p&gt;
&lt;p&gt;Poll too slowly and latency becomes terrible or the device&#39;s RX queue fills.&lt;/p&gt;
&lt;p&gt;Poll too aggressively and the driver wastes CPU time and occupies the SCSI bus.&lt;/p&gt;
&lt;p&gt;There is another constraint: effectively there is one command path. RX polling and TX must be arbitrated. If receive traffic monopolizes the SCSI transport, transmit suffers. If transmit is allowed to dominate, the receive queue may overflow.&lt;/p&gt;
&lt;p&gt;This becomes much more important once BlueSCSI is placed behind another bridge.&lt;/p&gt;
&lt;h2 id=&quot;native-scsi-was-the-easy-case&quot;&gt;Native SCSI was the easy case&lt;/h2&gt;
&lt;p&gt;With a real SCSI HBA, one Ethernet frame can be transferred in one SCSI operation.&lt;/p&gt;
&lt;p&gt;A typical receive record looks like this:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/rx-record.svg&quot; alt=&quot;DaynaPORT receive record&quot; /&gt;&lt;/p&gt;
&lt;p&gt;BlueSCSI can also return several records in one &lt;code&gt;READ(6)&lt;/code&gt; when the host uses the multi-packet form.&lt;/p&gt;
&lt;p&gt;The driver therefore cannot assume that one SCSI read equals one Ethernet frame. It has to walk the returned buffer record by record.&lt;/p&gt;
&lt;p&gt;For transmit, the simplest path is even cleaner: one raw Ethernet frame becomes one &lt;code&gt;WRITE(6)&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This is close to how the original hardware was intended to be used.&lt;/p&gt;
&lt;p&gt;Then came the USB bridge.&lt;/p&gt;
&lt;h2 id=&quot;the-512-byte-problem&quot;&gt;The 512-byte problem&lt;/h2&gt;
&lt;p&gt;My BlueSCSI bridge work introduced a very different transport constraint.&lt;/p&gt;
&lt;p&gt;The early USB Mass Storage bridge path moved SCSI data through a &lt;strong&gt;512-byte buffer&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;That is perfectly natural for block storage.&lt;/p&gt;
&lt;p&gt;It is not natural for Ethernet.&lt;/p&gt;
&lt;p&gt;A normal Ethernet frame can be roughly 1.5 KB, so a packet that was previously transferred in one SCSI operation suddenly no longer fits.&lt;/p&gt;
&lt;p&gt;The transport now looks more like this:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/bridge-512.svg&quot; alt=&quot;512-byte SCSI bridge transfer&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This is where the driver stopped being only a DaynaPORT driver and started becoming a DaynaPORT-over-bridge driver.&lt;/p&gt;
&lt;h2 id=&quot;adding-chunking-without-breaking-daynaport&quot;&gt;Adding chunking without breaking DaynaPORT&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/bridge-chunking.svg&quot; alt=&quot;An Ethernet frame split into continuation chunks and reassembled&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Bridge-specific fragmentation preserves the Ethernet frame despite a smaller transport limit.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The bridge protocol uses a continuation flag (&lt;code&gt;0x20&lt;/code&gt;) so that one Ethernet frame can be spread across multiple SCSI transactions.&lt;/p&gt;
&lt;p&gt;On receive, the driver collects chunks until it sees the final record and then reconstructs the Ethernet frame.&lt;/p&gt;
&lt;p&gt;On transmit, a large frame is split into several &lt;code&gt;WRITE(6)&lt;/code&gt; commands and reassembled on the BlueSCSI side.&lt;/p&gt;
&lt;p&gt;For the USB bridge path, TX chunks are limited to 506 bytes because some bytes are consumed by protocol framing.&lt;/p&gt;
&lt;p&gt;Conceptually:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/continuation.svg&quot; alt=&quot;DaynaPORT continuation flag&quot; /&gt;&lt;/p&gt;
&lt;p&gt;A zero-length write can be used to abort an incomplete transmit chain.&lt;/p&gt;
&lt;p&gt;The important architectural decision was to keep this extension outside the normal DaynaPORT semantics as much as possible. Native SCSI should continue to behave like native SCSI. Chunking is transport-specific.&lt;/p&gt;
&lt;p&gt;That distinction appears throughout the Linux and Windows implementations.&lt;/p&gt;
&lt;h2 id=&quot;the-linux-driver-became-three-drivers&quot;&gt;The Linux driver became three drivers&lt;/h2&gt;
&lt;p&gt;The original Linux driver work comes from Jeff Flitton. My fork adds the bridge/chunking support, diagnostics, shared protocol work, and subsequent improvements.&lt;/p&gt;
&lt;p&gt;One source file cannot realistically cover the Linux kernel API from 2.0 through 7.x, so the repository now contains three separate driver implementations:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;linux-2.0/&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;linux-2.4/&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;linux-7.0/&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What can be shared lives in:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;lib/daynaport.h&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That file deliberately contains protocol logic rather than Linux kernel API calls.&lt;/p&gt;
&lt;p&gt;It is C89-clean so that the same protocol core can be used by very old compilers and modern toolchains.&lt;/p&gt;
&lt;p&gt;This turned out to be one of the most valuable design choices in the project, because the same core could later be reused by the Windows ports.&lt;/p&gt;
&lt;h2 id=&quot;discovery-instead-of-hard-coded-scsi-ids&quot;&gt;Discovery instead of hard-coded SCSI IDs&lt;/h2&gt;
&lt;p&gt;Another small but important rule is that the driver must not assume a fixed SCSI target.&lt;/p&gt;
&lt;p&gt;BlueSCSI lets the user choose the SCSI ID of the emulated DaynaPORT.&lt;/p&gt;
&lt;p&gt;The driver therefore follows normal SCSI discovery and binds to a device matching the Dayna identity:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Peripheral type: 0x03&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Vendor:          Dayna&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Product:         SCSI/Link&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This sounds obvious, but it keeps the driver compatible with:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;real DaynaPORT hardware,&lt;/li&gt;
&lt;li&gt;BlueSCSI,&lt;/li&gt;
&lt;li&gt;ZuluSCSI,&lt;/li&gt;
&lt;li&gt;PiSCSI,&lt;/li&gt;
&lt;li&gt;and potentially more than one adapter on a bus.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;another-historical-detail-wait-after-enable&quot;&gt;Another historical detail: wait after ENABLE&lt;/h2&gt;
&lt;p&gt;One of the less obvious pieces of the protocol is the delay after enabling the interface.&lt;/p&gt;
&lt;p&gt;After:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;0E 00 00 00 00 80&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;the host should wait about 500 ms before issuing further commands.&lt;/p&gt;
&lt;p&gt;That behavior is visible in other DaynaPORT driver implementations as well.&lt;/p&gt;
&lt;p&gt;It is exactly the sort of detail that disappears if you only implement the obvious packet path and do not compare multiple implementations and the original command documentation.&lt;/p&gt;
&lt;p&gt;The protocol-reference work in the Linux repository became almost as important as the driver itself.&lt;/p&gt;
&lt;h2 id=&quot;polling-was-the-real-performance-problem&quot;&gt;Polling was the real performance problem&lt;/h2&gt;
&lt;p&gt;At first it was tempting to assume that the 512-byte bridge buffer would be the main bottleneck.&lt;/p&gt;
&lt;p&gt;The benchmarks showed something more interesting.&lt;/p&gt;
&lt;p&gt;The Linux test notes record roughly:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/polling-baseline.svg&quot; alt=&quot;Fixed-interval receive polling&quot; /&gt;&lt;/p&gt;
&lt;p&gt;During active receive traffic, the driver reached roughly 70–85 SCSI reads per second.&lt;/p&gt;
&lt;p&gt;That was close to the device round-trip limit in the tested configuration.&lt;/p&gt;
&lt;p&gt;The polling logic was changed so that after a productive read the driver could continue polling aggressively for a short fast window rather than immediately sleeping for the normal poll interval.&lt;/p&gt;
&lt;p&gt;When the link becomes idle, it falls back to the slower polling cadence.&lt;/p&gt;
&lt;p&gt;This is important because a fixed polling interval creates a nasty feedback loop:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/polling-adaptive.svg&quot; alt=&quot;Adaptive receive polling&quot; /&gt;&lt;/p&gt;
&lt;p&gt;With back-to-back packets, every artificial sleep directly reduces throughput.&lt;/p&gt;
&lt;p&gt;The optimized path instead behaves approximately like:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;packet arrives&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;productive READ&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;immediate READ&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;productive READ&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;immediate READ&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;...&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;empty READs&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;      |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;return to idle cadence&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The cost is temporarily higher CPU usage, so the driver exposes parameters such as &lt;code&gt;poll_ms&lt;/code&gt;, &lt;code&gt;poll0_ms&lt;/code&gt;, &lt;code&gt;fast_hold&lt;/code&gt;, and &lt;code&gt;tx_burst&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&quot;measuring-the-bridge-separately-changed-the-diagnosis&quot;&gt;Measuring the bridge separately changed the diagnosis&lt;/h2&gt;
&lt;p&gt;The most useful benchmark was not an Ethernet benchmark.&lt;/p&gt;
&lt;p&gt;It was block I/O through the same bridge.&lt;/p&gt;
&lt;p&gt;Using the raw bridge with a SCSI Jaz drive produced approximately:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;block read:   ~2.7 Mbit/s&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;block write:  ~2.6 Mbit/s&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That was several times faster than the DaynaPORT network download result.&lt;/p&gt;
&lt;p&gt;This immediately showed that the USB bridge itself had significant headroom.&lt;/p&gt;
&lt;p&gt;The network RX path was being limited somewhere else.&lt;/p&gt;
&lt;p&gt;That is exactly why end-to-end benchmarking alone can be misleading. If I had only measured &lt;code&gt;iperf3&lt;/code&gt;, it would have been easy to blame USB chunking for everything.&lt;/p&gt;
&lt;h2 id=&quot;increasing-the-usb-buffer-to-4096-bytes&quot;&gt;Increasing the USB buffer to 4096 bytes&lt;/h2&gt;
&lt;p&gt;A later experiment in the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/BlueSCSI-v2/tree/avkghost/birdge-with-daynaport&quot;&gt;BlueSCSI DaynaPORT branch&lt;/a&gt; increased the TinyUSB MSC buffer from 512 bytes to 4096 bytes and used burst-oriented transfers. The Linux path was updated to take advantage of the larger/burst-capable transport as well.&lt;/p&gt;
&lt;p&gt;That changed block-read performance significantly.&lt;/p&gt;
&lt;p&gt;The benchmark notes show approximately:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;512-byte era:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    block read  ~2.7 Mbit/s&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;4096-byte buffer:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    block read  ~5.35 Mbit/s&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A full Ethernet frame could now fit into one bridge read instead of being split across up to three 512-byte transactions.&lt;/p&gt;
&lt;p&gt;A code audit of the branch confirms that this is implemented in the device
side and bridge paths, not just described in the benchmark notes. The firmware
sets the TinyUSB MSC endpoint buffer to 4096 bytes. Its block-device initiator
also contains read-ahead/prefetch logic, adaptive prefetch depth, USB locking,
and dead-target recovery. Separately, the DaynaPORT bridge path limits each
MSC response to the available buffer, reassembles oversized frames, and can
batch complete frames in blind/multi-packet mode.&lt;/p&gt;
&lt;p&gt;Those code paths establish the available capability of the modified BlueSCSI
branch. They do not prove that a particular Windows block-device driver
requested or exercised the larger-buffer or burst behavior; that still
requires a Windows-side trace or reproducible benchmark.&lt;/p&gt;
&lt;p&gt;But something else was revealing: the Jaz drive itself was now the write-side
limit.&lt;/p&gt;
&lt;p&gt;And the expected DaynaPORT network improvement was small, because the Wi-Fi inbound path was already the tighter bottleneck.&lt;/p&gt;
&lt;p&gt;That is the kind of result I like in low-level projects: changing a parameter makes one subsystem dramatically faster while proving that another subsystem was never limited by that parameter in the first place.&lt;/p&gt;
&lt;h2 id=&quot;from-linux-to-windows&quot;&gt;From Linux to Windows&lt;/h2&gt;
&lt;p&gt;Once the DaynaPORT protocol logic had been separated from the Linux kernel glue, porting the concept to Windows became much more realistic. The validated Linux implementation is the behavioral reference for the Windows port; the Windows-specific KMDF, NDIS, INF, and signing layers still have to be checked against their Microsoft toolchain.&lt;/p&gt;
&lt;p&gt;But &quot;Windows driver&quot; does not mean one architecture.&lt;/p&gt;
&lt;p&gt;The repository currently targets four eras.&lt;/p&gt;
&lt;h3 id=&quot;windows-10-11&quot;&gt;Windows 10 / 11&lt;/h3&gt;
&lt;p&gt;Modern Windows uses a two-driver split:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;daynaportfn.sys&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    KMDF companion driver&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    owns the SCSI-side communication&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;daynaportnd.sys&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    NDIS 6 miniport&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    exposes the network adapter&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/windows-modern.svg&quot; alt=&quot;Windows 10/11 driver layers&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The modern Windows split keeps the NDIS-facing network adapter separate from
the KMDF SCSI companion.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;This keeps SCSI access and NDIS networking separated.&lt;/p&gt;
&lt;h3 id=&quot;windows-2000-xp&quot;&gt;Windows 2000 / XP&lt;/h3&gt;
&lt;p&gt;The 2000/XP version uses a single NDIS 5.1 miniport.&lt;/p&gt;
&lt;p&gt;It talks directly to the SCSI device through the NT storage stack, rather than
using the modern KMDF companion split.&lt;/p&gt;
&lt;h3 id=&quot;windows-nt-4-0&quot;&gt;Windows NT 4.0&lt;/h3&gt;
&lt;p&gt;NT 4 follows the same broad idea but targets NDIS 4.0 and the NT-era driver environment.&lt;/p&gt;
&lt;h3 id=&quot;windows-95-98-me&quot;&gt;Windows 95 / 98 / ME&lt;/h3&gt;
&lt;p&gt;Windows 9x is different again.&lt;/p&gt;
&lt;p&gt;The design uses an NDIS 4 miniport together with a companion SCSI VxD. The VxD
is also the least-validated part of the Windows source tree and still contains
an ASPI dispatch stub.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/windows-9x.svg&quot; alt=&quot;Windows 9x driver layers&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The amusing part is that the network protocol itself did not fundamentally change.&lt;/p&gt;
&lt;p&gt;The operating-system plumbing changed completely.&lt;/p&gt;
&lt;h2 id=&quot;one-protocol-core-decades-of-driver-apis&quot;&gt;One protocol core, decades of driver APIs&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/cross-platform-drivers.svg&quot; alt=&quot;Linux and Windows driver implementations using a common DaynaPORT protocol core&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Platform-specific driver plumbing stays separate from the protocol core.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/common-core.svg&quot; alt=&quot;Shared protocol across operating systems&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The shared protocol layer isolates framing and chunking from OS-specific APIs.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The Windows repository vendors the shared DaynaPORT protocol header derived from the Linux project.&lt;/p&gt;
&lt;p&gt;That means the project can be viewed as two layers:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+-------------------------------------------+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| OS-specific driver                       |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;|                                           |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| Linux 2.0 / 2.4 / 7.x                    |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| Win9x / NT4 / 2K-XP / Win10-11           |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+--------------------+----------------------+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                     |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                     v&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+-------------------------------------------+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| shared DaynaPORT protocol core            |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| framing, opcodes, RX parsing, chunking    |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+--------------------+----------------------+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                     |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                     v&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+-------------------------------------------+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;| SCSI transport / BlueSCSI                 |&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;+-------------------------------------------+&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is probably the most reusable part of the whole exercise.&lt;/p&gt;
&lt;p&gt;The protocol is old.&lt;/p&gt;
&lt;p&gt;The driver APIs are not.&lt;/p&gt;
&lt;p&gt;Keeping those two facts separate makes the code much easier to reason about.&lt;/p&gt;
&lt;h2 id=&quot;bridge-options-on-windows&quot;&gt;Bridge options on Windows&lt;/h2&gt;
&lt;p&gt;The Windows side follows the same split between normal DaynaPORT traffic and bridge-specific transport.&lt;/p&gt;
&lt;p&gt;The legacy configuration includes tunables for:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/project-timeline.svg&quot; alt=&quot;Evolution of the project&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;UsbBridge&lt;/code&gt; selects the USB MSC chunked path.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;RawBridge&lt;/code&gt; selects the BlueSCSI raw DaynaPORT bridge path, where both directions must respect the bridge transfer limit.&lt;/p&gt;
&lt;p&gt;The important point is that the network stack should not care.&lt;/p&gt;
&lt;p&gt;It still sends and receives Ethernet frames.&lt;/p&gt;
&lt;p&gt;Only the SCSI transport layer knows whether a frame was moved in one command or reconstructed from several.&lt;/p&gt;
&lt;h2 id=&quot;what-the-repository-history-shows&quot;&gt;What the repository history shows&lt;/h2&gt;
&lt;p&gt;Looking across the repositories, the project did not evolve as one giant &quot;write a driver&quot; commit.&lt;/p&gt;
&lt;p&gt;It evolved in layers:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;1. Understand the DaynaPORT command set&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;2. Build a Linux driver around normal SCSI&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;3. Separate reusable protocol logic&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;4. Add multi-kernel Linux implementations&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;5. Add USB-bridge chunking&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;6. Add diagnostics and benchmark tooling&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;7. Tune the polling loop&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;8. Benchmark storage and network paths independently&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;9. Increase the MSC buffer and measure again&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;10. Reuse the protocol core for Windows&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;11. Split Windows support by driver era&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This progression matters because several design decisions only became obvious after measurement.&lt;/p&gt;
&lt;p&gt;For example, the 512-byte bridge limitation looked like the obvious performance problem.&lt;/p&gt;
&lt;p&gt;It was a real limitation — but not the dominant limitation for Wi-Fi receive.&lt;/p&gt;
&lt;p&gt;Likewise, making the poll loop faster improved behavior, but only until the device round-trip became the next ceiling.&lt;/p&gt;
&lt;h2 id=&quot;current-status&quot;&gt;Current status&lt;/h2&gt;
&lt;p&gt;The Linux repository is the most mature part of the project.&lt;/p&gt;
&lt;p&gt;The modified BlueSCSI device has also been confirmed to operate stably with
Windows, without the crashes seen during earlier development. This validates
the device-side bridge and its Windows block-device integration at that stage;
it does not establish that the Windows block-device driver exercised the
larger-buffer or burst path, and it should not be read as validation of the
separate Windows DaynaPORT host driver.&lt;/p&gt;
&lt;p&gt;It contains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Linux 2.0.x support&lt;/li&gt;
&lt;li&gt;Linux 2.4.x support&lt;/li&gt;
&lt;li&gt;Linux 7.x support&lt;/li&gt;
&lt;li&gt;shared DaynaPORT protocol code&lt;/li&gt;
&lt;li&gt;USB bridge chunking&lt;/li&gt;
&lt;li&gt;diagnostics&lt;/li&gt;
&lt;li&gt;a benchmark runbook&lt;/li&gt;
&lt;li&gt;protocol documentation&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The Windows repository currently contains source implementations for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Windows 10/11&lt;/li&gt;
&lt;li&gt;Windows 2000/XP&lt;/li&gt;
&lt;li&gt;Windows NT 4.0&lt;/li&gt;
&lt;li&gt;Windows 9x&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All four Windows trees contain era-specific source implementations and
source-level lifecycle, error-handling, and transport corrections. They still
have different validation status: none has a reproducible Microsoft WDK/DDK
build, installation log, or hardware-test record committed in the repository.
The Win9x tree has an additional explicit limitation: its ASPI VxD dispatch is
still a stub and is not ready for hardware validation.&lt;/p&gt;
&lt;p&gt;Most importantly, the Windows repository still needs reproducible validation
with the corresponding Microsoft toolchains and hardware/emulator
combinations. There are no committed Windows binaries, catalogs, build logs, or
Windows-driver CI runs in the repository. The existing CI run is limited to the
portable protocol-core tests. That is evidence about the repository, not proof
that no earlier external build or installation ever succeeded.&lt;/p&gt;
&lt;p&gt;The Win9x tree should be treated more cautiously than the other Windows trees:
its ASPI VxD dispatch still contains an explicit stub and requires additional
DDK and hardware work.&lt;/p&gt;
&lt;p&gt;I consider that distinction important: source code that looks complete is not the same thing as a validated driver.&lt;/p&gt;
&lt;p&gt;The Windows repository keeps the current evidence boundary in its
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-windows-driver/blob/main/docs/validation-matrix.md&quot;&gt;validation matrix&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;what-i-learned&quot;&gt;What I learned&lt;/h2&gt;
&lt;p&gt;A few lessons from this project are useful beyond BlueSCSI.&lt;/p&gt;
&lt;h3 id=&quot;storage-bridges-and-packet-protocols-have-different-assumptions&quot;&gt;Storage bridges and packet protocols have different assumptions&lt;/h3&gt;
&lt;p&gt;A 512-byte transfer buffer is perfectly reasonable for a storage bridge.&lt;/p&gt;
&lt;p&gt;It becomes a protocol-design problem when the payload is a 1500-byte Ethernet frame.&lt;/p&gt;
&lt;h3 id=&quot;polling-behavior-can-dominate-throughput&quot;&gt;Polling behavior can dominate throughput&lt;/h3&gt;
&lt;p&gt;When there is no receive interrupt, a few milliseconds of unnecessary sleep can matter more than the raw bus bandwidth.&lt;/p&gt;
&lt;h3 id=&quot;measure-layers-independently&quot;&gt;Measure layers independently&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;iperf3&lt;/code&gt; told me the network was slow.&lt;/p&gt;
&lt;p&gt;Block I/O through the same bridge showed that USB itself was much faster.&lt;/p&gt;
&lt;p&gt;That changed where I looked for the bottleneck.&lt;/p&gt;
&lt;h3 id=&quot;keep-protocol-code-separate-from-os-apis&quot;&gt;Keep protocol code separate from OS APIs&lt;/h3&gt;
&lt;p&gt;The shared C89 DaynaPORT core made it possible to support Linux kernels separated by decades and then reuse the same logic in Windows drivers with completely different architectures.&lt;/p&gt;
&lt;h3 id=&quot;old-protocols-are-often-better-documented-in-code-than-in-manuals&quot;&gt;Old protocols are often better documented in code than in manuals&lt;/h3&gt;
&lt;p&gt;The useful details came from several places:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the original Dayna command reference,&lt;/li&gt;
&lt;li&gt;BlueSCSI/ZuluSCSI firmware,&lt;/li&gt;
&lt;li&gt;PiSCSI,&lt;/li&gt;
&lt;li&gt;existing BSD driver behavior,&lt;/li&gt;
&lt;li&gt;and real benchmark results.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;No single source tells the whole story.&lt;/p&gt;
&lt;h2 id=&quot;final-result&quot;&gt;Final result&lt;/h2&gt;
&lt;p&gt;My original motivation was to connect physical SCSI hardware — including the Sony SMO-F551 and Iomega Jaz 1 GB / 2 GB drives — when a suitable vintage SCSI controller was not available. BlueSCSI Ultra provided the starting point for that experiment.&lt;/p&gt;
&lt;p&gt;The original DaynaPORT was a SCSI-to-Ethernet adapter designed for computers of the early 1990s.&lt;/p&gt;
&lt;p&gt;BlueSCSI already proved that this protocol can be emulated.&lt;/p&gt;
&lt;p&gt;The next step was to turn that emulation around and make Linux and Windows machines treat it as a network interface — including when the SCSI traffic itself has to cross another 512-byte USB bridge. Linux reached a working, tested implementation; Windows remains a documented port whose WDK build, installation, and hardware validation are still separate tasks.&lt;/p&gt;
&lt;p&gt;That required more than wrapping &lt;code&gt;READ(6)&lt;/code&gt; and &lt;code&gt;WRITE(6)&lt;/code&gt; in a network driver.&lt;/p&gt;
&lt;p&gt;It required:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;protocol reconstruction,&lt;/li&gt;
&lt;li&gt;SCSI discovery,&lt;/li&gt;
&lt;li&gt;polling and arbitration,&lt;/li&gt;
&lt;li&gt;packet chunking and reassembly,&lt;/li&gt;
&lt;li&gt;cross-kernel Linux work,&lt;/li&gt;
&lt;li&gt;several generations of Windows driver architecture,&lt;/li&gt;
&lt;li&gt;and a lot of benchmarking to determine which bottleneck was actually real.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That is what made the project interesting.&lt;/p&gt;
&lt;p&gt;The network protocol is more than thirty years old.&lt;/p&gt;
&lt;p&gt;The engineering problems are not.&lt;/p&gt;
&lt;h2 id=&quot;repositories&quot;&gt;Repositories&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;BlueSCSI v2 experimental branch: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/BlueSCSI-v2/tree/avkghost/birdge-with-daynaport&quot;&gt;https://github.com/avkghost/BlueSCSI-v2/tree/avkghost/birdge-with-daynaport&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Linux DaynaPORT driver: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-linux-driver&quot;&gt;https://github.com/avkghost/daynaport-scsilink-linux-driver&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Linux bridge branch: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-linux-driver/tree/avkghost/birdge-with-daynaport&quot;&gt;https://github.com/avkghost/daynaport-scsilink-linux-driver/tree/avkghost/birdge-with-daynaport&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Windows DaynaPORT drivers: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-windows-driver&quot;&gt;https://github.com/avkghost/daynaport-scsilink-windows-driver&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Linux DaynaPORT protocol notes: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-linux-driver/blob/main/reference/daynaport.md&quot;&gt;https://github.com/avkghost/daynaport-scsilink-linux-driver/blob/main/reference/daynaport.md&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Linux bridge benchmark notes: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/daynaport-scsilink-linux-driver/blob/main/reference/benchmark.md&quot;&gt;https://github.com/avkghost/daynaport-scsilink-linux-driver/blob/main/reference/benchmark.md&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Recovering Iomega Media in 2026: Building iomega_tools for Jaz, Zip, and QIC-122</title>
        <published>2026-08-09T00:00:00+00:00</published>
        <updated>2026-08-09T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/iomega-tools/"/>
        <id>https://iamavk.info/blog/iomega-tools/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/iomega-tools/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/hero.png&quot; alt=&quot;Abstract illustration of recovering vintage removable media&quot; /&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; This post was created on August 9, 2026 and later expanded as the tooling and hardware tests evolved during August.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The story started with a simple problem: I still had useful SCSI hardware, but no convenient way to connect it to a modern machine.&lt;/p&gt;
&lt;p&gt;That problem eventually led me to experiment with a BlueSCSI Ultra as a bridge to physical SCSI devices. I described that part of the project in &lt;a href=&quot;/blog/bluescsi-daynaport-linux-windows/&quot;&gt;From BlueSCSI Ultra to DaynaPORT: Connecting Legacy SCSI Devices to Linux and Windows&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Among the devices I was testing were Iomega Jaz drives.&lt;/p&gt;
&lt;p&gt;Once the transport problem was mostly solved, another question appeared:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;What can I actually do with old Iomega media once Linux can see the drive?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Mounting a filesystem is only part of the answer.&lt;/p&gt;
&lt;p&gt;Old removable storage has its own failure modes, protection states, formatting behavior, vendor commands, damaged sectors, strange bridge interactions, and—in the case of old Iomega backup software—archive formats that modern systems do not understand.&lt;/p&gt;
&lt;p&gt;That became the reason for &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/iomega_tools&quot;&gt;iomega_tools&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;why-iomega-media-is-still-interesting&quot;&gt;Why Iomega media is still interesting&lt;/h2&gt;
&lt;p&gt;Zip and Jaz are good examples of a period when removable storage was not just a generic USB block device.&lt;/p&gt;
&lt;p&gt;At the host level, the drives may look familiar:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/scsi-stack.svg&quot; alt=&quot;Scsi stack&quot; /&gt;&lt;/p&gt;
&lt;p&gt;But there is more going on below the filesystem.&lt;/p&gt;
&lt;p&gt;A drive can have:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;media protection modes,&lt;/li&gt;
&lt;li&gt;password-based locking,&lt;/li&gt;
&lt;li&gt;vendor-specific status pages,&lt;/li&gt;
&lt;li&gt;mechanical eject and load behavior,&lt;/li&gt;
&lt;li&gt;FORMAT UNIT behavior,&lt;/li&gt;
&lt;li&gt;firmware-specific command restrictions,&lt;/li&gt;
&lt;li&gt;bad sectors that stall a bridge instead of returning a clean SCSI error,&lt;/li&gt;
&lt;li&gt;and media layouts created by old Iomega software.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A modern Linux kernel can handle the ordinary block-device part.&lt;/p&gt;
&lt;p&gt;For recovery work, however, I wanted to see and control the layer underneath it.&lt;/p&gt;
&lt;h2 id=&quot;the-project&quot;&gt;The project&lt;/h2&gt;
&lt;p&gt;The repository currently builds three independent tools:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/utilities.svg&quot; alt=&quot;Utilities&quot; /&gt;&lt;/p&gt;
&lt;p&gt;There was also an SG_IO tracing path during the reverse-engineering work.&lt;/p&gt;
&lt;p&gt;The code is intentionally implemented as normal Linux userspace tooling. There is no custom kernel driver for Jaz or Zip. Commands are sent through the Linux SCSI generic interface using &lt;code&gt;SG_IO&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That makes the project useful with more than one transport:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/transport.svg&quot; alt=&quot;Transport&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This was important for my setup because BlueSCSI Ultra was already being used as a path from a modern host to physical legacy SCSI hardware.&lt;/p&gt;
&lt;h2 id=&quot;starting-with-scsi-passthrough&quot;&gt;Starting with SCSI passthrough&lt;/h2&gt;
&lt;p&gt;The first layer is deliberately boring: identify the device, send commands, decode sense data, and avoid hiding failures.&lt;/p&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./scsi_probe&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; identify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./scsi_probe&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; explore&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;scsi_probe&lt;/code&gt; is read-only. I use it when I want to understand what a device accepts before adding behavior to the main tool.&lt;/p&gt;
&lt;p&gt;This became useful beyond Iomega itself. The probe code eventually grew better MMC decoding and DVD structure reporting as I kept using it against other SCSI-class devices.&lt;/p&gt;
&lt;p&gt;But the original purpose was much simpler: find out what the Iomega hardware actually does.&lt;/p&gt;
&lt;h2 id=&quot;reading-files-without-trusting-the-host-mount-path&quot;&gt;Reading files without trusting the host mount path&lt;/h2&gt;
&lt;p&gt;For FAT-family media, &lt;code&gt;iomega_tools&lt;/code&gt; can access the filesystem directly through FatFs.&lt;/p&gt;
&lt;p&gt;That allows operations such as:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;r&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; get&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; README.TXT&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; get&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-all&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;o&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; recovered&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;get&lt;/code&gt; path supports FAT12, FAT16, FAT32 and exFAT directly.&lt;/p&gt;
&lt;p&gt;Long filenames and subdirectories are supported, and filesystem detection does not assume that the filesystem starts at one hard-coded sector.&lt;/p&gt;
&lt;p&gt;This matters because Iomega media can contain an MBR layout rather than a filesystem beginning at LBA 0.&lt;/p&gt;
&lt;p&gt;The code recognizes both:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/filesystem-layout.svg&quot; alt=&quot;Filesystem layout&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The traditional Iomega layout prefers partition table slot 4, but the tools do not require it.&lt;/p&gt;
&lt;h2 id=&quot;recover-first-repair-later&quot;&gt;Recover first, repair later&lt;/h2&gt;
&lt;p&gt;One of the main design rules is that inspection should be read-only unless I explicitly ask the program to modify the cartridge.&lt;/p&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; verify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;verify&lt;/code&gt; combines several checks:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;drive and protection-state probing,&lt;/li&gt;
&lt;li&gt;filesystem consistency checking where a safe read-only checker exists,&lt;/li&gt;
&lt;li&gt;and a full read-only surface scan unless it is disabled.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Likewise:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; surface&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;is read-only by default.&lt;/p&gt;
&lt;p&gt;This is the behavior I want from a recovery tool. Before changing anything on old media, I want to know what can still be read.&lt;/p&gt;
&lt;h2 id=&quot;when-a-bad-sector-is-not-just-a-bad-sector&quot;&gt;When a bad sector is not just a bad sector&lt;/h2&gt;
&lt;p&gt;Old hardware behind a modern bridge produces failure modes that are easy to miss if software assumes a normal SATA or USB disk.&lt;/p&gt;
&lt;p&gt;A large read may fail even though most of the requested sectors are fine.&lt;/p&gt;
&lt;p&gt;The surface scanner therefore works in stages:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/surface-scan.svg&quot; alt=&quot;Surface scan&quot; /&gt;&lt;/p&gt;
&lt;p&gt;After a chunk failure, the scanner can fall back to single-sector reads for the remaining media.&lt;/p&gt;
&lt;p&gt;This turned out to matter with USB-to-SCSI/IDE paths. One observed bridge configuration rejected large &lt;code&gt;READ(10)&lt;/code&gt; transfers at roughly 240 sectors.&lt;/p&gt;
&lt;p&gt;The tooling therefore cannot simply assume that a failed transfer means every sector inside that transfer is bad.&lt;/p&gt;
&lt;h2 id=&quot;recovering-from-a-stalled-bridge&quot;&gt;Recovering from a stalled bridge&lt;/h2&gt;
&lt;p&gt;There is an even less pleasant case: a failing data phase can leave the transport itself in a bad state.&lt;/p&gt;
&lt;p&gt;Instead of receiving a nice SCSI CHECK CONDITION, the host may get an ioctl-level transport error.&lt;/p&gt;
&lt;p&gt;The recovery path became:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/bridge-recovery.svg&quot; alt=&quot;Bridge recovery&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/bridge-recovery-sequence.svg&quot; alt=&quot;Iomega bridge recovery sequence: transport error, reset, optional re-enumeration, and resume&quot; /&gt;&lt;/p&gt;
&lt;p&gt;For USB storage, the reset can become a USB device reset so the bridge toggles the underlying drive reset line.&lt;/p&gt;
&lt;p&gt;If the device node actually disappears with &lt;code&gt;ENODEV&lt;/code&gt;, the scanner can force a USB re-enumeration and resume.&lt;/p&gt;
&lt;p&gt;That sounds excessive for ordinary storage.&lt;/p&gt;
&lt;p&gt;For recovering aging removable media through an adapter stack, it is extremely useful.&lt;/p&gt;
&lt;h2 id=&quot;capacity-without-read-capacity&quot;&gt;Capacity without READ CAPACITY&lt;/h2&gt;
&lt;p&gt;Another interesting hardware detail appeared while working with Zip.&lt;/p&gt;
&lt;p&gt;Normally, a disk utility would ask a SCSI disk for its size using &lt;code&gt;READ CAPACITY&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That assumption does not always work here.&lt;/p&gt;
&lt;p&gt;The Zip drive I tested can reject &lt;code&gt;READ CAPACITY&lt;/code&gt; with an illegal-request sense response.&lt;/p&gt;
&lt;p&gt;So when the user does not specify the scan length, &lt;code&gt;surface&lt;/code&gt; can determine the readable limit by probing the address space instead of depending entirely on the normal capacity command.&lt;/p&gt;
&lt;p&gt;It is a small feature, but it captures an important theme of the project:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;old hardware often works correctly according to its own assumptions, not ours.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;bad-sector-registries-instead-of-a-private-database&quot;&gt;Bad-sector registries instead of a private database&lt;/h2&gt;
&lt;p&gt;Finding damaged sectors is useful.&lt;/p&gt;
&lt;p&gt;Preventing a filesystem from reusing them can be even more useful.&lt;/p&gt;
&lt;p&gt;With an explicit:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; surface&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-mark-bad&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;the tool can write bad locations into the filesystem&#39;s own bad-block mechanism.&lt;/p&gt;
&lt;p&gt;The implementation is filesystem-specific:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/bad-block-mapping.svg&quot; alt=&quot;Bad block mapping&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Metadata areas and already allocated data are protected from this operation.&lt;/p&gt;
&lt;p&gt;Formats without an appropriate bad-block registry are refused.&lt;/p&gt;
&lt;p&gt;This is intentionally not the default. &lt;code&gt;--mark-bad&lt;/code&gt; modifies the media and requires write access.&lt;/p&gt;
&lt;h2 id=&quot;iomega-protection-modes&quot;&gt;Iomega protection modes&lt;/h2&gt;
&lt;p&gt;Another part of the project came from reverse-engineering IomegaWare behavior.&lt;/p&gt;
&lt;p&gt;The drives support more than a simple writable/read-only switch.&lt;/p&gt;
&lt;p&gt;The tool implements protection operations including:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;normal read/write,&lt;/li&gt;
&lt;li&gt;write protection,&lt;/li&gt;
&lt;li&gt;password-protected write access,&lt;/li&gt;
&lt;li&gt;password-based read/write lock,&lt;/li&gt;
&lt;li&gt;temporary unprotect,&lt;/li&gt;
&lt;li&gt;password removal.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; password&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;w&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; password&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;rw&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;pw&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; secret&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; password&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;u&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;pw&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; secret&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The password is carried in the vendor command parameter list, matching the behavior reconstructed from Iomega&#39;s old Linux tooling.&lt;/p&gt;
&lt;p&gt;The same research also informed &lt;code&gt;mount&lt;/code&gt;, &lt;code&gt;umount&lt;/code&gt;, &lt;code&gt;eject&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt;, and &lt;code&gt;info&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&quot;formatting-was-more-complicated-than-expected&quot;&gt;Formatting was more complicated than expected&lt;/h2&gt;
&lt;p&gt;At first glance, formatting a Jaz or Zip cartridge sounds like:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/logical-format.svg&quot; alt=&quot;Logical format&quot; /&gt;&lt;/p&gt;
&lt;p&gt;That is only the logical side.&lt;/p&gt;
&lt;p&gt;The tool can create an Iomega-style MBR layout and invoke the appropriate Linux &lt;code&gt;mkfs&lt;/code&gt; implementation:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; format&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-fs&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; fat16&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Supported filesystem paths include FAT variants, exFAT, NTFS and ext filesystems.&lt;/p&gt;
&lt;p&gt;But IomegaWare also has a drive-level long-format path.&lt;/p&gt;
&lt;p&gt;That path sends a SCSI &lt;code&gt;FORMAT UNIT&lt;/code&gt; command and then monitors the drive while the physical format is in progress.&lt;/p&gt;
&lt;p&gt;Reverse-engineering the old Linux IomegaWare binaries exposed the command form and the progress calculation used by the original software.&lt;/p&gt;
&lt;p&gt;Conceptually:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/long-format.svg&quot; alt=&quot;Long format&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The distinction between logical formatting and a real drive-level format turned out to be important.&lt;/p&gt;
&lt;h2 id=&quot;a-bluescsi-specific-surprise-sense-data-can-disappear&quot;&gt;A BlueSCSI-specific surprise: sense data can disappear&lt;/h2&gt;
&lt;p&gt;This was one of the more interesting interactions between the Iomega project and my BlueSCSI work.&lt;/p&gt;
&lt;p&gt;During a long format, the drive reports:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/format-sense.svg&quot; alt=&quot;Format sense&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The format progress value lives in bytes 16-17 of fixed-format sense data.&lt;/p&gt;
&lt;p&gt;The confusing observation was that BlueSCSI debug output could show the expected &lt;code&gt;04/04&lt;/code&gt; sense data, while a later host-side &lt;code&gt;REQUEST SENSE&lt;/code&gt; returned &lt;code&gt;00/00/00&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The repository notes explain why this is plausible.&lt;/p&gt;
&lt;p&gt;In initiator mode, BlueSCSI can issue &lt;code&gt;REQUEST SENSE&lt;/code&gt; internally after a command returns CHECK CONDITION. The bridge consumes and acts on that sense data, but it does not necessarily preserve it for a later &lt;code&gt;REQUEST SENSE&lt;/code&gt; from the Linux host.&lt;/p&gt;
&lt;p&gt;So the path can look like this:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/consumed-sense.svg&quot; alt=&quot;Consumed sense&quot; /&gt;&lt;/p&gt;
&lt;p&gt;That changed the implementation.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;TEST UNIT READY&lt;/code&gt; became the reliable completion condition, while &lt;code&gt;REQUEST SENSE&lt;/code&gt; is used for progress information when it is available.&lt;/p&gt;
&lt;p&gt;This is exactly the kind of issue that appears only when several old and new protocol layers are stacked together.&lt;/p&gt;
&lt;h2 id=&quot;what-iomegaware-was-still-useful-for&quot;&gt;What IomegaWare was still useful for&lt;/h2&gt;
&lt;p&gt;I did not want to copy the original tooling.&lt;/p&gt;
&lt;p&gt;I wanted to recover protocol facts.&lt;/p&gt;
&lt;p&gt;The old IomegaWare for Linux binaries were therefore used as reverse-engineering reference material.&lt;/p&gt;
&lt;p&gt;The work exposed details such as:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the drive-level &lt;code&gt;FORMAT UNIT&lt;/code&gt; command,&lt;/li&gt;
&lt;li&gt;the 4-byte format parameter list,&lt;/li&gt;
&lt;li&gt;progress calculation from REQUEST SENSE,&lt;/li&gt;
&lt;li&gt;which drive classes are allowed to perform a long format,&lt;/li&gt;
&lt;li&gt;protection state encoding,&lt;/li&gt;
&lt;li&gt;the vendor protection command,&lt;/li&gt;
&lt;li&gt;mount/unmount/eject sequencing.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The implementation in this repository is original C/C++ code built from those observed protocol behaviors.&lt;/p&gt;
&lt;p&gt;This distinction matters both technically and legally: old proprietary binaries are reference material, not project dependencies.&lt;/p&gt;
&lt;h2 id=&quot;qic-extract-recovering-old-iomega-backup-archives&quot;&gt;&lt;code&gt;qic_extract&lt;/code&gt;: recovering old Iomega backup archives&lt;/h2&gt;
&lt;p&gt;The third program in the repository solves a different problem.&lt;/p&gt;
&lt;p&gt;Even if the disk itself is readable, old backup software may have stored the useful data inside a QIC-122 archive.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;qic_extract&lt;/code&gt; operates on an image or archive file rather than directly on the SCSI device:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;./qic_extract&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; analyze&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; backup.qic&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;./qic_extract&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; backup.qic&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;./qic_extract&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; extract&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; backup.qic&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;o&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; recovered&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The reverse-engineered format contains structures such as:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/qic-structure.svg&quot; alt=&quot;Qic structure&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The compressed path includes a custom bitstream decoder, segment-chain handling, and UTF-16 filename recovery.&lt;/p&gt;
&lt;p&gt;I find this part of the project particularly interesting because it shows that recovering an old cartridge can involve several independent layers:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/recovery-layers.svg&quot; alt=&quot;Recovery layers&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Being able to read the disk is not the same as being able to recover the files that a user originally stored.&lt;/p&gt;
&lt;h2 id=&quot;the-commit-history-tells-the-real-story&quot;&gt;The commit history tells the real story&lt;/h2&gt;
&lt;p&gt;The repository was created on August 9, 2026 with the three main tools already separated:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/initial-tools.svg&quot; alt=&quot;Initial tools&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The first commit also included the SCSI passthrough layer, filesystem work, QIC extraction, tests, and the SG_IO tracing infrastructure.&lt;/p&gt;
&lt;p&gt;Later the same day the project gained more practical recovery features: image/benchmark streaming, wildcard file recovery, additional filesystem formatting support, and ATA SMART probing through SAT.&lt;/p&gt;
&lt;p&gt;The next rounds of work came directly from hardware behavior:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/timeline.svg&quot; alt=&quot;Timeline&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This is one reason I like keeping experimental work in git.&lt;/p&gt;
&lt;p&gt;The interesting history is not just &quot;feature A was added.&quot;&lt;/p&gt;
&lt;p&gt;It shows which assumptions failed when the code met real hardware.&lt;/p&gt;
&lt;h2 id=&quot;the-relationship-with-the-bluescsi-project&quot;&gt;The relationship with the BlueSCSI project&lt;/h2&gt;
&lt;p&gt;The BlueSCSI work and &lt;code&gt;iomega_tools&lt;/code&gt; solve different halves of the same problem.&lt;/p&gt;
&lt;p&gt;BlueSCSI Ultra gave me a practical way to connect physical legacy SCSI hardware when I did not have a suitable vintage host adapter.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;iomega_tools&lt;/code&gt; deals with what happens after that connection exists.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/bluescsi-connection.svg&quot; alt=&quot;Bluescsi connection&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The DaynaPORT project pushed the bridge in another direction: transporting Ethernet frames over SCSI.&lt;/p&gt;
&lt;p&gt;The Iomega project pushes it toward storage recovery and hardware diagnostics.&lt;/p&gt;
&lt;p&gt;Together they turned BlueSCSI Ultra from something I originally associated with SCSI emulation into a much more interesting laboratory for real legacy hardware.&lt;/p&gt;
&lt;h2 id=&quot;current-workflow&quot;&gt;Current workflow&lt;/h2&gt;
&lt;p&gt;For an unknown old cartridge, my preferred workflow is conservative.&lt;/p&gt;
&lt;p&gt;First identify the drive:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./scsi_probe&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; identify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then inspect the media:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; info&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;r&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If the filesystem is readable, recover the files:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; get&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-all&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;o&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; recovered&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then perform a read-only check:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./iomega_tools&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; verify&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; /dev/sdX&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;For questionable media, perform an explicit surface scan or image it before attempting repair.&lt;/p&gt;
&lt;p&gt;Only after recovery would I consider write operations such as formatting or marking bad blocks.&lt;/p&gt;
&lt;p&gt;That ordering is intentional:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/iomega-tools/recovery-workflow.svg&quot; alt=&quot;Recovery workflow&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Old removable media is not the place for optimistic automation.&lt;/p&gt;
&lt;h2 id=&quot;what-i-learned&quot;&gt;What I learned&lt;/h2&gt;
&lt;p&gt;A few conclusions from the project are broader than Iomega.&lt;/p&gt;
&lt;p&gt;First, &lt;strong&gt;SCSI passthrough is still extremely useful&lt;/strong&gt;. A modern kernel provides the infrastructure needed to investigate devices that were designed decades ago.&lt;/p&gt;
&lt;p&gt;Second, &lt;strong&gt;a bridge is part of the protocol path&lt;/strong&gt;. It can impose transfer limits, transform errors, consume sense data, reset hardware, or disappear during a failed command.&lt;/p&gt;
&lt;p&gt;Third, &lt;strong&gt;filesystem recovery and device recovery are different problems&lt;/strong&gt;. A perfectly valid FAT parser cannot help if the transport stalls on one physical sector.&lt;/p&gt;
&lt;p&gt;Fourth, &lt;strong&gt;old vendor tools are valuable documentation&lt;/strong&gt; when the real protocol documentation is incomplete or difficult to find.&lt;/p&gt;
&lt;p&gt;And finally, &lt;strong&gt;the safest recovery feature is usually the one that does not write anything&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;That is why &lt;code&gt;iomega_tools&lt;/code&gt; started as a collection of low-level experiments and gradually became a recovery toolkit.&lt;/p&gt;
&lt;p&gt;The code is available in the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/iomega_tools&quot;&gt;iomega_tools repository&lt;/a&gt;.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Building Cisco-Like Loopback Interfaces on Linux</title>
        <published>2026-01-12T00:00:00+00:00</published>
        <updated>2026-01-12T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/linux-cisco-like-loopback/"/>
        <id>https://iamavk.info/blog/linux-cisco-like-loopback/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/linux-cisco-like-loopback/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/hero.png&quot; alt=&quot;Abstract illustration of Linux loopback networking and policy routes&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Cisco routers use loopback interfaces as stable Layer 3 identities. They are independent of physical ports, making them useful for router IDs, routing-protocol endpoints, management addresses, and service IPs. Linux can provide nearly the same functionality without a custom kernel module: use a &lt;strong&gt;dummy&lt;/strong&gt; network interface.&lt;/p&gt;
&lt;p&gt;This article explains how dummy interfaces behave, how to route to them, how to use them with policy routing and FRRouting, and how to create one from C using Netlink. It also covers anycast, VRRP, and the cases where a custom kernel driver or a TUN device is actually necessary.&lt;/p&gt;
&lt;h2 id=&quot;1-dummy-versus-loopback-what-is-actually-routable&quot;&gt;1. Dummy versus loopback: what is actually routable?&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/dummy-routing.svg&quot; alt=&quot;Dummy Routing&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Create a dummy device and assign a host address:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; link&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; type&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; addr&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dev&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; link&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; set&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; up&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; addr&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; show&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Linux installs a &lt;strong&gt;local&lt;/strong&gt; route for the address, normally visible in &lt;code&gt;table local&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; show&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; table&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; local&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; local 10.255.0.1 dev dummy0 proto kernel scope host src 10.255.0.1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;ping 10.255.0.1&lt;/code&gt; succeeds on the host. This does &lt;strong&gt;not&lt;/strong&gt; mean that packets are physically transmitted through &lt;code&gt;dummy0&lt;/code&gt;: packets destined to a local address are delivered by the IP stack. The dummy driver&#39;s transmit path discards packets sent &lt;em&gt;out&lt;/em&gt; through it.&lt;/p&gt;
&lt;p&gt;To reach the address from another machine, that machine or its upstream routers must know a path to the Linux host. For example, if the Linux host has &lt;code&gt;eth0=192.168.1.1&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; On a second machine on 192.168.1.0/24:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; via&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 192.168.1.1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ping&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 10.255.0.1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The packet arrives on &lt;code&gt;eth0&lt;/code&gt; and is delivered locally because &lt;code&gt;10.255.0.1&lt;/code&gt; belongs to the host. &lt;strong&gt;IPv4 forwarding is not required&lt;/strong&gt; for this traffic. Forwarding is required only for packets whose destinations are &lt;em&gt;not&lt;/em&gt; local and which must traverse the router.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;A dummy interface provides a routable local identity, not a virtual wire. Adding &lt;code&gt;ip route add 10.200.0.0/24 dev dummy0&lt;/code&gt; does not make &lt;code&gt;10.200.0.0/24&lt;/code&gt; reachable: packets routed out of the dummy device are dropped.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id=&quot;2-source-based-policy-routing&quot;&gt;2. Source-based policy routing&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/policy-routing.svg&quot; alt=&quot;Policy Routing&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Suppose a router has these addresses:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Device&lt;/th&gt;&lt;th&gt;Address&lt;/th&gt;&lt;th&gt;Role&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;dummy0&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;10.255.0.1/32&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Stable identity&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;eth0&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;192.168.1.1/24&lt;/code&gt;&lt;/td&gt;&lt;td&gt;LAN&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;eth1&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;172.16.0.2/24&lt;/code&gt;&lt;/td&gt;&lt;td&gt;ISP 1&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;eth2&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;172.17.0.2/24&lt;/code&gt;&lt;/td&gt;&lt;td&gt;ISP 2&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;To route traffic &lt;em&gt;sourced from&lt;/em&gt; the dummy address through ISP 2:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; default&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; via&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 172.17.0.1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dev&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; eth2&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; table&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 200&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; rule&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; from&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; lookup&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 200&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; priority&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 1000&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; rule&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; show&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; get&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 8.8.8.8&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; from&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 10.255.0.1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The expected lookup selects &lt;code&gt;eth2&lt;/code&gt;, with source &lt;code&gt;10.255.0.1&lt;/code&gt;. This only works end-to-end if the upstream network accepts the source address and a return route exists. On multihomed hosts, reverse-path filtering (&lt;code&gt;rp_filter&lt;/code&gt;) can also interfere with asymmetric paths.&lt;/p&gt;
&lt;p&gt;Linux checks the local-routing rule before ordinary user policy rules, so traffic destined to a locally owned dummy address is ordinarily delivered locally.&lt;/p&gt;
&lt;h2 id=&quot;3-stable-bgp-endpoints-with-frrouting&quot;&gt;3. Stable BGP endpoints with FRRouting&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/bgp-peering.svg&quot; alt=&quot;Bgp Peering&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Loopback addresses make good BGP peering endpoints because the session can survive a change in physical path &lt;em&gt;provided another routed path remains available&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Example topology:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;&lt;/th&gt;&lt;th&gt;Router A&lt;/th&gt;&lt;th&gt;Router B&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Physical IP&lt;/td&gt;&lt;td&gt;&lt;code&gt;192.168.10.1&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;192.168.10.2&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Dummy IP&lt;/td&gt;&lt;td&gt;&lt;code&gt;10.255.0.1/32&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;10.255.0.2/32&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;ASN&lt;/td&gt;&lt;td&gt;&lt;code&gt;65001&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;65002&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;On each router, create the appropriate dummy interface. Before starting the BGP session, provide underlay reachability:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Router A&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.255.0.2/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; via&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 192.168.10.2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6A737D, #6A737D);&quot;&gt; Router B&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; route&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; via&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 192.168.10.1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Install FRRouting (&lt;code&gt;sudo apt install frr&lt;/code&gt; on Debian/Ubuntu), enable &lt;code&gt;bgpd&lt;/code&gt; in &lt;code&gt;/etc/frr/daemons&lt;/code&gt;, and configure it with &lt;code&gt;vtysh&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Router A:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;configure terminal&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;router bgp 65001&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; bgp router-id 10.255.0.1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.2 remote-as 65002&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.2 update-source 10.255.0.1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.2 ebgp-multihop 2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; address-family ipv4 unicast&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  neighbor 10.255.0.2 activate&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; exit-address-family&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Router B:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;configure terminal&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;router bgp 65002&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; bgp router-id 10.255.0.2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.1 remote-as 65001&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.1 update-source 10.255.0.2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; neighbor 10.255.0.1 ebgp-multihop 2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; address-family ipv4 unicast&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  neighbor 10.255.0.1 activate&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; exit-address-family&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Check the session with:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; vtysh&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;#39;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;show bgp summary&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;A BGP router ID is only a 32-bit identifier: it need not be assigned to an interface or reachable. Using the loopback address as the router ID is a convention, not a requirement.&lt;/p&gt;
&lt;h2 id=&quot;4-anycast-service-addresses&quot;&gt;4. Anycast service addresses&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/anycast.svg&quot; alt=&quot;Anycast&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Anycast uses the &lt;strong&gt;same service IP on multiple hosts&lt;/strong&gt;, with the routing system selecting which host receives traffic. For example, three DNS servers might all own &lt;code&gt;10.100.0.53/32&lt;/code&gt; on &lt;code&gt;dummy0&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; link&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; type&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; addr&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; add&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; 10.100.0.53/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dev&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; link&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; set&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; dummy0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; up&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Each server advertises the prefix via its routing daemon. In FRRouting, the advertisement can include:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;router bgp 65001&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; address-family ipv4 unicast&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  network 10.100.0.53/32&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; exit-address-family&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This assumes BGP neighbors are configured and the prefix exists in the relevant routing table. Routing preference is determined by routing policies and metrics; it is &lt;strong&gt;not necessarily geographic proximity&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;A key failure mode is that the dummy interface stays up even when the DNS application fails. A production deployment must link route advertisement or withdrawal to service health. Anycast is particularly useful for stateless services such as DNS; stateful TCP sessions can break if routing shifts an active flow to another node.&lt;/p&gt;
&lt;h2 id=&quot;5-vrrp-and-high-availability&quot;&gt;5. VRRP and high availability&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/vrrp.svg&quot; alt=&quot;Vrrp&quot; /&gt;&lt;/p&gt;
&lt;p&gt;For a virtual IP on the same Ethernet subnet as clients, Keepalived/VRRP is often a better fit than an always-present dummy address. Consider:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Server A: &lt;code&gt;192.168.1.10&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Server B: &lt;code&gt;192.168.1.11&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Floating VIP: &lt;code&gt;192.168.1.100&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Example Keepalived configuration for Server A:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;vrrp_instance VI_1 {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    state BACKUP&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    interface eth0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    virtual_router_id 51&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    priority 150&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    advert_int 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    virtual_ipaddress {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        192.168.1.100/24 dev eth0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Server B uses compatible settings with a lower priority, for example &lt;code&gt;100&lt;/code&gt;. For a VIP on a directly attached subnet, assigning it to &lt;code&gt;eth0&lt;/code&gt; supports normal neighbor discovery. For &lt;strong&gt;routed&lt;/strong&gt; service IPs, a dummy interface plus dynamic routing may be preferable. Keepalived configurations also need appropriate peer communication, health checks, and security controls for production use.&lt;/p&gt;
&lt;h2 id=&quot;6-create-a-dummy-interface-from-c-with-libnl&quot;&gt;6. Create a dummy interface from C with libnl&lt;/h2&gt;
&lt;p&gt;The preferred implementation is a userspace management program using &lt;strong&gt;rtnetlink&lt;/strong&gt;, not a new kernel network driver. Netlink messages include:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Task&lt;/th&gt;&lt;th&gt;Message&lt;/th&gt;&lt;th&gt;Key attributes&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Create interface&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_NEWLINK&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;IFLA_IFNAME&lt;/code&gt;, &lt;code&gt;IFLA_LINKINFO&lt;/code&gt;, &lt;code&gt;IFLA_INFO_KIND=dummy&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Bring up&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_NEWLINK&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;IFF_UP&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Add IP address&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_NEWADDR&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;IFA_LOCAL&lt;/code&gt;, &lt;code&gt;IFA_ADDRESS&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Add route&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_NEWROUTE&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTA_DST&lt;/code&gt;, &lt;code&gt;RTA_GATEWAY&lt;/code&gt;, &lt;code&gt;RTA_OIF&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Add policy rule&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_NEWRULE&lt;/code&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;FRA_SRC&lt;/code&gt;, &lt;code&gt;FRA_TABLE&lt;/code&gt;, &lt;code&gt;FRA_PRIORITY&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Delete interface&lt;/td&gt;&lt;td&gt;&lt;code&gt;RTM_DELLINK&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Interface identifier&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Install development packages:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; apt&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; install&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; libnl-3-dev&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; libnl-route-3-dev&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; pkg-config&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Minimal implementation (&lt;code&gt;cloop.c&lt;/code&gt;):&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;c&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;stdio.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;net/if.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;sys/socket.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/netlink.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/socket.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/route/link.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/route/link/dummy.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/route/addr.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;#&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;include&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;netlink/addr.h&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;int&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; main&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;void&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    struct&lt;/span&gt;&lt;span&gt; nl_sock &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;*&lt;/span&gt;&lt;span&gt;sock &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; nl_socket_alloc&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    struct&lt;/span&gt;&lt;span&gt; rtnl_link &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;*&lt;/span&gt;&lt;span&gt;link &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; NULL&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; *&lt;/span&gt;&lt;span&gt;change &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; NULL&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    struct&lt;/span&gt;&lt;span&gt; rtnl_addr &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;*&lt;/span&gt;&lt;span&gt;addr &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; NULL&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    struct&lt;/span&gt;&lt;span&gt; nl_addr &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;*&lt;/span&gt;&lt;span&gt;ip &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; NULL&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    int&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;1&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; index&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;!&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; return&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 1&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; nl_connect&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; NETLINK_ROUTE&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    link &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_dummy_alloc&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;!&lt;/span&gt;&lt;span&gt;link&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; -&lt;/span&gt;&lt;span&gt;NLE_NOMEM&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;    rtnl_link_set_name&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;link&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;cloop0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_add&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; link&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; NLM_F_CREATE &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;|&lt;/span&gt;&lt;span&gt; NLM_F_EXCL&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    index &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_name2i&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;cloop0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;index &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; -&lt;/span&gt;&lt;span&gt;NLE_OBJ_NOTFOUND&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    addr &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_addr_alloc&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;!&lt;/span&gt;&lt;span&gt;addr&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; -&lt;/span&gt;&lt;span&gt;NLE_NOMEM&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; nl_addr_parse&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; AF_INET&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; &amp;amp;&lt;/span&gt;&lt;span&gt;ip&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;    rtnl_addr_set_ifindex&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;addr&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; index&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_addr_set_local&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;addr&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; ip&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_addr_add&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; addr&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    change &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_alloc&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;!&lt;/span&gt;&lt;span&gt;change&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt; {&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; -&lt;/span&gt;&lt;span&gt;NLE_NOMEM&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;span&gt; }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;    rtnl_link_set_flags&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;change&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; IFF_UP&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_change&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; link&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span&gt; change&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; goto&lt;/span&gt;&lt;span&gt; out&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;    puts&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Created cloop0: 10.255.0.1/32&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;out:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; fprintf&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;stderr&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;Netlink error: &lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;%s&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;\n&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; nl_geterror&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;err&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;change&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_put&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;change&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;ip&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; nl_addr_put&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;ip&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;addr&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_addr_put&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;addr&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    if&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;link&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt; rtnl_link_put&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;link&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;    nl_socket_free&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;sock&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;    return&lt;/span&gt;&lt;span&gt; err &lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; ?&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color: light-dark(#D73A49, #F97583);&quot;&gt; :&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 0&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Compile and run:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;gcc&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; cloop.c&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;o&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; cloop&lt;/span&gt;&lt;span&gt; $(&lt;/span&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;pkg-config&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-cflags&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;-libs&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; libnl-3.0&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; libnl-route-3.0&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; ./cloop&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt;d&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; link&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; show&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; cloop0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ip&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; addr&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; show&lt;/span&gt;&lt;span style=&quot;color: light-dark(#032F62, #9ECBFF);&quot;&gt; cloop0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: light-dark(#6F42C1, #B392F0);&quot;&gt;ping&lt;/span&gt;&lt;span style=&quot;color: light-dark(#005CC5, #79B8FF);&quot;&gt; 10.255.0.1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The program requires &lt;code&gt;CAP_NET_ADMIN&lt;/code&gt; in its network namespace. It intentionally uses &lt;code&gt;NLM_F_EXCL&lt;/code&gt;, so rerunning it while &lt;code&gt;cloop0&lt;/code&gt; exists returns an error. This is a minimal demonstration: a production daemon should reconcile desired state, handle Netlink notifications, validate inputs, and roll back partially applied changes.&lt;/p&gt;
&lt;h2 id=&quot;7-do-you-need-a-kernel-module&quot;&gt;7. Do you need a kernel module?&lt;/h2&gt;
&lt;p&gt;Usually not. A kernel module using &lt;code&gt;alloc_netdev()&lt;/code&gt; and &lt;code&gt;register_netdev()&lt;/code&gt; can expose a new device, but it introduces kernel API compatibility and lifecycle concerns. A device built with &lt;code&gt;ether_setup()&lt;/code&gt; is Ethernet-like, not an exact clone of Linux &lt;code&gt;lo&lt;/code&gt; (&lt;code&gt;ARPHRD_LOOPBACK&lt;/code&gt; with loopback-specific semantics).&lt;/p&gt;
&lt;p&gt;Use existing device types unless you truly need new transmit/receive behavior:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Requirement&lt;/th&gt;&lt;th&gt;Recommended mechanism&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Stable router IP / Cisco-style Loopback&lt;/td&gt;&lt;td&gt;&lt;code&gt;dummy&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Custom source-based routing&lt;/td&gt;&lt;td&gt;&lt;code&gt;ip rule&lt;/code&gt; / rtnetlink&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;BGP/OSPF control plane&lt;/td&gt;&lt;td&gt;FRRouting&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Service anycast&lt;/td&gt;&lt;td&gt;&lt;code&gt;dummy&lt;/code&gt; + BGP + health checks&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Active/passive floating LAN VIP&lt;/td&gt;&lt;td&gt;Keepalived / VRRP&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Userspace processing of IP packets&lt;/td&gt;&lt;td&gt;TUN&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Virtual Ethernet frames in userspace&lt;/td&gt;&lt;td&gt;TAP&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Routed path between hosts&lt;/td&gt;&lt;td&gt;GRE, IPIP, WireGuard, etc.&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Novel device-specific kernel TX/RX semantics&lt;/td&gt;&lt;td&gt;Custom &lt;code&gt;net_device&lt;/code&gt; driver&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;h2 id=&quot;8-suggested-architecture-for-a-router-like-product&quot;&gt;8. Suggested architecture for a router-like product&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/linux-cisco-like-loopback/c-architecture.svg&quot; alt=&quot;C Architecture&quot; /&gt;&lt;/p&gt;
&lt;p&gt;A clean Cisco-like management architecture has three layers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;C management daemon:&lt;/strong&gt; parses commands such as &lt;code&gt;interface Loopback0&lt;/code&gt;, stores desired configuration, creates dummy interfaces, assigns addresses, and watches rtnetlink notifications.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Linux kernel:&lt;/strong&gt; owns the interfaces, local and forwarding routing tables, policy rules, and packet forwarding.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;FRRouting:&lt;/strong&gt; implements BGP/OSPF, route advertisements, and convergence.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For example, a CLI command &lt;code&gt;interface Loopback0&lt;/code&gt; could map to &lt;code&gt;dummy&lt;/code&gt; device &lt;code&gt;lo0&lt;/code&gt; or &lt;code&gt;Loopback0&lt;/code&gt;; &lt;code&gt;ip address 10.255.0.1/32&lt;/code&gt; maps to &lt;code&gt;RTM_NEWADDR&lt;/code&gt;; and routing protocol configuration is delegated to FRRouting.&lt;/p&gt;
&lt;p&gt;This approach preserves the operational advantages of Cisco loopbacks without maintaining a custom kernel driver. The central rule is simple: &lt;strong&gt;dummy interfaces anchor addresses; physical interfaces and tunnels carry transit traffic.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;references&quot;&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://man7.org/linux/man-pages/man8/ip-link.8.html&quot;&gt;Linux &lt;code&gt;ip-link(8)&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://man7.org/linux/man-pages/man8/ip-route.8.html&quot;&gt;Linux &lt;code&gt;ip-route(8)&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://man7.org/linux/man-pages/man8/ip-rule.8.html&quot;&gt;Linux &lt;code&gt;ip-rule(8)&lt;/code&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://man7.org/linux/man-pages/man7/rtnetlink.7.html&quot;&gt;Linux rtnetlink(7)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://docs.frrouting.org/&quot;&gt;FRRouting documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.infradead.org/~tgr/libnl/&quot;&gt;libnl documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a rel=&quot;external&quot; href=&quot;https://www.keepalived.org/&quot;&gt;Keepalived documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Restoring a Roland MT-120: Building an Arduino Floppy Drive Tester</title>
        <published>2025-08-25T00:00:00+00:00</published>
        <updated>2025-08-25T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/roland-mt120-fdtester/"/>
        <id>https://iamavk.info/blog/roland-mt120-fdtester/</id>
        
        <content type="html" xml:base="https://iamavk.info/blog/roland-mt120-fdtester/">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/hero.png&quot; alt=&quot;Abstract illustration of restoring a vintage drum machine with a floppy tester&quot; /&gt;&lt;/p&gt;
&lt;p&gt;An old Roland MT-120 was the starting point for an unexpectedly interesting embedded project. I wanted to restore the instrument, but its floppy drive was not working correctly. Before replacing parts or making mechanical adjustments, I needed a way to control the drive independently and see what its signals were doing.&lt;/p&gt;
&lt;p&gt;That led to &lt;strong&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester&quot;&gt;FDTester&lt;/a&gt;&lt;/strong&gt;: an open-source Arduino-based tester for standard PC-style 3.5-inch floppy drives. Rather than attempting to read files, it exercises the drive&#39;s mechanical and electrical interface directly.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Scope:&lt;/strong&gt; FDTester is a mechanical and electrical diagnostic tool, &lt;strong&gt;not&lt;/strong&gt; a complete floppy disk controller. It does not decode MFM sectors, read filesystem data, or prove that the magnetic read channel works.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id=&quot;the-problem-a-drive-that-could-not-read-other-disks&quot;&gt;The problem: a drive that could not read other disks&lt;/h2&gt;
&lt;p&gt;The Roland MT-120 restoration raised a practical question: was the failure caused by the drive mechanics, its interface signals, the spindle speed, or head positioning? In particular, &lt;strong&gt;Track 0 alignment&lt;/strong&gt; was a challenge: the drive did not reliably read disks written by other devices.&lt;/p&gt;
&lt;p&gt;A drive can respond to motor and seek commands while still being unable to read a disk correctly. A Track 0 sensor confirms that the mechanism reached its reference position; it does not, by itself, prove correct radial head alignment or interchangeability with another drive. That distinction shaped the test strategy.&lt;/p&gt;
&lt;h2 id=&quot;architecture-separating-control-from-the-host&quot;&gt;Architecture: separating control from the host&lt;/h2&gt;
&lt;p&gt;FDTester uses an Arduino as a small, dedicated interface controller. A serial terminal sends commands; the Arduino drives the floppy interface and reports status and measurements. The repository documents an &lt;strong&gt;Arduino Uno / ATmega328P&lt;/strong&gt; reference setup, while its project description also mentions Mega compatibility.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/architecture.svg&quot; alt=&quot;FDTester system architecture: serial terminal, Arduino, buffered control lines, floppy drive and external power&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The repository&#39;s validated target is an &lt;strong&gt;Epson SMD-300&lt;/strong&gt;, configured for DS0. This is the documented test configuration; compatibility with other drives depends on their pinout, jumpers, and electrical characteristics.&lt;/p&gt;
&lt;p&gt;The floppy drive has its &lt;strong&gt;own external 5 V supply&lt;/strong&gt; (the README specifies approximately 1 A capability), with a common ground shared with the Arduino. Do not power the motor from the Arduino USB port. A ULN2803 or appropriate transistor buffering is recommended for the active-low interface lines; check your exact drive&#39;s electrical requirements before connecting it.&lt;/p&gt;
&lt;h2 id=&quot;the-signals-that-matter&quot;&gt;The signals that matter&lt;/h2&gt;
&lt;p&gt;A classic 34-pin floppy interface gives direct access to the most useful diagnostic controls and feedback signals:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/interface.svg&quot; alt=&quot;Simplified drive interface and documented Arduino Uno pin mapping&quot; /&gt;&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Floppy pin&lt;/th&gt;&lt;th&gt;Signal&lt;/th&gt;&lt;th&gt;Arduino Uno pin&lt;/th&gt;&lt;th&gt;Diagnostic purpose&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;10&lt;/td&gt;&lt;td&gt;Drive Select 0&lt;/td&gt;&lt;td&gt;D7&lt;/td&gt;&lt;td&gt;Select the target drive&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;16&lt;/td&gt;&lt;td&gt;Motor Enable&lt;/td&gt;&lt;td&gt;D6&lt;/td&gt;&lt;td&gt;Start and stop spindle&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;18&lt;/td&gt;&lt;td&gt;Direction&lt;/td&gt;&lt;td&gt;D5&lt;/td&gt;&lt;td&gt;Set seek direction&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;20&lt;/td&gt;&lt;td&gt;Step&lt;/td&gt;&lt;td&gt;D4&lt;/td&gt;&lt;td&gt;Advance the head by one track&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;32&lt;/td&gt;&lt;td&gt;Side Select&lt;/td&gt;&lt;td&gt;D8&lt;/td&gt;&lt;td&gt;Select head 0 or 1&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;8&lt;/td&gt;&lt;td&gt;Index&lt;/td&gt;&lt;td&gt;D2 / INT0&lt;/td&gt;&lt;td&gt;Count revolutions&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;26&lt;/td&gt;&lt;td&gt;Track 0&lt;/td&gt;&lt;td&gt;D11&lt;/td&gt;&lt;td&gt;Detect home position&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;28&lt;/td&gt;&lt;td&gt;Write Protect&lt;/td&gt;&lt;td&gt;D12&lt;/td&gt;&lt;td&gt;Inspect disk protection status&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;34&lt;/td&gt;&lt;td&gt;Disk Change&lt;/td&gt;&lt;td&gt;D10&lt;/td&gt;&lt;td&gt;Detect media change (default configuration)&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;24&lt;/td&gt;&lt;td&gt;Write Gate&lt;/td&gt;&lt;td&gt;D9&lt;/td&gt;&lt;td&gt;Dangerous write control; locked by default&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;&lt;strong&gt;Important:&lt;/strong&gt; The table reflects the repository&#39;s Epson SMD-300 / DS0 wiring. Odd-numbered connector pins are grounds. Pin 34 may have a different function on other drives or jumper settings. Pin 2 (Density Select) is left unconnected in the documented setup.&lt;/p&gt;
&lt;h2 id=&quot;diagnostic-workflow&quot;&gt;Diagnostic workflow&lt;/h2&gt;
&lt;p&gt;I wanted a repeatable way to distinguish an interface problem from a mechanical problem before attempting fine adjustment.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/workflow.svg&quot; alt=&quot;Suggested step-by-step diagnostic workflow, including Track 0 verification and separate alignment checks&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The basic sequence is straightforward:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Check wiring and drive selection.&lt;/strong&gt; Confirm common ground, the external supply, and the DS0 jumper configuration.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Spin the motor.&lt;/strong&gt; Confirm that the mechanism starts reliably.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Measure rotational speed.&lt;/strong&gt; Count Index pulses over a known interval.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Home the head.&lt;/strong&gt; Move toward Track 0 and inspect the Track 0 signal.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Exercise stepping.&lt;/strong&gt; Seek to known tracks and sweep across a range to reveal mechanical issues.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Inspect status lines.&lt;/strong&gt; Check Write Protect and Disk Change while inserting or removing media.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Evaluate disk interchangeability separately.&lt;/strong&gt; If disks from other devices still cannot be read, investigate mechanical alignment and the read path using appropriate reference media and equipment.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;rpm-measurement-using-the-index-pulse&quot;&gt;RPM measurement using the Index pulse&lt;/h3&gt;
&lt;p&gt;The Index signal provides one pulse per revolution on a typical drive. If &lt;code&gt;N&lt;/code&gt; pulses arrive during a measurement window of &lt;code&gt;T&lt;/code&gt; seconds, rotational speed is approximately:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;RPM = 60 × N / T&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;For example, 15 pulses over three seconds corresponds to about 300 RPM. That is a calculation example, &lt;strong&gt;not&lt;/strong&gt; a recorded measurement from the Roland restoration.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/rpm.svg&quot; alt=&quot;Index pulses and a three-second measurement window for a 300 RPM example&quot; /&gt;&lt;/p&gt;
&lt;h3 id=&quot;track-0-homing-is-not-alignment&quot;&gt;Track 0: homing is not alignment&lt;/h3&gt;
&lt;p&gt;The most interesting part of this repair was Track 0. Homing tells the firmware where the mechanism&#39;s reference sensor is triggered. However, an incorrectly positioned head assembly can still produce a plausible Track 0 signal and fail to read disks recorded by another machine.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/roland-mt120-fdtester/track0.svg&quot; alt=&quot;Conceptual distinction between Track 0 sensor detection and physical head alignment&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This is why I treated &lt;strong&gt;Track 0 detection&lt;/strong&gt; and &lt;strong&gt;head alignment&lt;/strong&gt; as different problems. FDTester can help with the former and with repeatable movement tests; it cannot directly measure magnetic head alignment or validate sector decoding. Alignment should be checked against appropriate reference media or specialist equipment rather than inferred solely from the Track 0 switch.&lt;/p&gt;
&lt;h2 id=&quot;using-fdtester&quot;&gt;Using FDTester&lt;/h2&gt;
&lt;p&gt;The firmware exposes a simple &lt;strong&gt;115200-baud serial console&lt;/strong&gt;. Once the drive is wired and powered correctly, these are useful commands from the published README:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color-scheme: light dark; color: light-dark(#24292E, #E1E4E8); background-color: light-dark(#FFFFFF, #24292E);&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;P                  # Report status&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;M                  # Toggle motor&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;R 3                # Measure RPM for 3 seconds&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;H                  # Home to Track 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;G 40               # Seek to absolute track 40&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;S 1                # Select side 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;T 0 79 100         # Sweep tracks 0–79, 100 ms dwell&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;RATE 6             # Set stepping interval to 6 ms&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;?                  # Show help&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The firmware also includes Write Gate controls protected by an explicit &lt;code&gt;UNLOCK&lt;/code&gt; command. &lt;strong&gt;Do not unlock or assert Write Gate on a disk containing valuable data.&lt;/strong&gt; None of the diagnostic steps above requires writing to the disk.&lt;/p&gt;
&lt;p&gt;See the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester/blob/main/FDTester.ino&quot;&gt;source sketch&lt;/a&gt; and &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester#readme&quot;&gt;README&lt;/a&gt; for the implementation and full command reference.&lt;/p&gt;
&lt;h2 id=&quot;what-the-tester-can-and-cannot-tell-you&quot;&gt;What the tester can—and cannot—tell you&lt;/h2&gt;
&lt;p&gt;FDTester is useful for isolating failures in spindle control, stepping, drive selection, home sensing, status wiring, and basic timing. It also gives you a repeatable way to move the mechanism while inspecting its behavior.&lt;/p&gt;
&lt;p&gt;It does &lt;strong&gt;not&lt;/strong&gt; establish that the read head, analog read channel, MFM decoding, or media are good. If the drive responds correctly to all commands but cannot read disks from other devices, the next step is targeted read-path and alignment diagnosis, not an assumption that the controller has fixed the problem.&lt;/p&gt;
&lt;h2 id=&quot;lessons-learned&quot;&gt;Lessons learned&lt;/h2&gt;
&lt;p&gt;The most valuable outcome was not a new storage controller; it was &lt;strong&gt;observability&lt;/strong&gt;. Breaking the drive into independent signals and commands turned a vague &quot;doesn&#39;t read disks&quot; failure into a sequence of testable questions.&lt;/p&gt;
&lt;p&gt;For me, the Roland MT-120 restoration was also a reminder of why embedded engineering is rewarding: software, timing, electrical interfaces, and mechanics all meet at the same debugging bench.&lt;/p&gt;
&lt;h2 id=&quot;related-projects&quot;&gt;Related projects&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/bluescsi-daynaport-linux-windows/&quot;&gt;From BlueSCSI Ultra to DaynaPORT&lt;/a&gt;: the same kind of hands-on work on legacy SCSI devices, from the controller side.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://iamavk.info/blog/iomega-tools/&quot;&gt;Recovering Iomega Media in 2026&lt;/a&gt;: the floppy-era hardware story continues with Jaz and Zip drives and QIC-122 tape.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Project:&lt;/strong&gt; &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester&quot;&gt;FDTester on GitHub&lt;/a&gt; · &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester/blob/main/FDTester.ino&quot;&gt;Arduino sketch&lt;/a&gt; · &lt;a rel=&quot;external&quot; href=&quot;https://github.com/avkghost/FDTester/blob/main/LICENSE&quot;&gt;GPL-3.0 license&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Diagrams in this article are conceptual illustrations, not oscilloscope captures or a certified wiring schematic. Confirm connections against the repository and your drive&#39;s technical manual.&lt;/em&gt;&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Welcome to the AVKSoft blog</title>
        <published>2025-08-01T00:00:00+00:00</published>
        <updated>2025-08-01T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://iamavk.info/blog/welcome/"/>
        <id>https://iamavk.info/blog/welcome/</id>
        
        <summary type="html">&lt;p&gt;&lt;img src=&quot;https://iamavk.info/blog/welcome/hero.png&quot; alt=&quot;Abstract illustration of an independent software maker building applications&quot; /&gt;&lt;/p&gt;
&lt;p&gt;AVKSoft is an independent software developer. We build mobile applications for the Apple App Store and Google Play.&lt;/p&gt;
&lt;p&gt;Our first applications are still in development. When they are ready, we will publish their store links on this site and announce them here.&lt;/p&gt;</summary>
        
    </entry>
</feed>
