The pipeline lives in odnelazm-ingest, a Rust crate that handles scraping, extraction, storage, and AI enrichment.

The source

Kenya’s Hansard is published through mzalendo.com as structured HTML. Each sitting is a flat document of sections, subsections, and contributions with speaker names. The odnelazm core library parses these into typed Rust structs. odnelazm-ingest takes those structs and writes them to PostgreSQL.

The pipeline

Everything runs through IngestPipeline, which is generic over any DataStore implementation. It holds a scraper, an optional embedder, an optional summarizer, and an optional metrics sink.

pub struct IngestPipeline<S: DataStore> {
    scraper: HansardScraper,
    store: S,
    embedder: Option<Arc<dyn Embedder>>,
    pub summarizer: Option<Arc<dyn Summarizer>>,
    pub metrics: Option<Arc<dyn MetricsSink>>,
}

The public surface is small: ingest_all_sittings, ingest_sittings_in_range, ingest_members, and ingest_member_profiles. Everything else is private.

What happens when a sitting is ingested

ingest_sitting is the core unit of work. For each sitting:

1. Sitting: upserted by URL. The full transcript is stored as JSONB in raw_json. On conflict, the raw JSON is overwritten but existing summaries are preserved.

2. Speakers: every distinct contributor name becomes a speakers row. Speaker names are extracted as-is from the HTML, which means the same person can appear under multiple names across sittings (“Hon. Kaluma”, “Peter Kaluma”, “Hon. Peter Kaluma”).

3. Bills: the bill extractor scans section headings for patterns ending in “BILL” or “ACT”. Each match becomes a bill_mentions row joining a bills record to the sitting, with stage detected from the heading or contribution text. Each speaker who contributed to that section is linked via bill_mention_speakers, with their full contribution text stored for enrichment.

4. Topics: “topic” is a generic term used across the codebase for any non-bill discussion item: questions and statements, notices of motion, and communications from the chair. These are identified by section type rather than heading pattern. Each topic is linked to its contributors via topic_speakers, again with full contribution text stored for enrichment.

Because bills are linked across sittings rather than stored per-sitting, this schema makes it possible to reconstruct a bill’s full legislative journey: every sitting it appeared in, the stage at each appearance, and everyone who spoke. Bunge Hub surfaces this as an interactive timeline on each bill page. You can trace a bill across sittings, seeing who spoke at each appearance and reading AI-generated summaries of each debate. Stages are inferred from section headings in the transcript so they are a close approximation rather than a guaranteed reflection of the formal legislative record.

The linking problem

After sittings are ingested, member profiles are imported from mzalendo’s member performance tracker. This gives us canonical names, constituencies, parties, and profile URLs.

Linking speakers to members is a three-step pass:

1. URL matching: when mzalendo includes a profile link on a contribution, exact URL match is used. Most reliable. This accounts for 1,871 of 3,021 speaker records in the current dataset.

2. Fuzzy name matching: for speakers without a URL, pg_trgm trigram similarity is computed against all member names via a match_member SQL function. Matches above a 0.45 score are accepted.

3. Role-based matching: presiding officers like “Hon. Speaker” and “Hon. Deputy Speaker” don’t fuzzy-match well because the names are too generic. These are resolved by looking up members with role = 'Speaker' or role ILIKE '%deputy speaker%' scoped to the relevant house and parliament.

Steps 2 and 3 combined account for a further 722 matches. 428 speaker records remain unmatched. Most are extraction noise where speech content leaked into the speaker name field during parsing. Others are genuinely ambiguous: presiding officers recorded under role-only labels like “Hon. Chairlady” that are too generic to resolve to a specific member without additional context.

The match_member function drives step 2. It cleans the raw speaker name, then scores it against every member using both word_similarity and similarity, taking whichever is higher:

CREATE OR REPLACE FUNCTION match_member(
    query_name TEXT,
    min_score  FLOAT DEFAULT 0.3
)
RETURNS TABLE (id UUID, name TEXT, url TEXT, house TEXT, constituency TEXT, score FLOAT)
LANGUAGE sql STABLE AS $$
    WITH cleaned AS (
        SELECT clean_speaker_name(query_name) AS cn
    )
    SELECT
        m.id, m.name, m.url, m.house, m.constituency,
        greatest(
            word_similarity(c.cn, m.name),
            similarity(c.cn, m.name)
        )::FLOAT AS score
    FROM members m, cleaned c
    WHERE greatest(
              word_similarity(c.cn, m.name),
              similarity(c.cn, m.name)
          ) >= min_score
    ORDER BY score DESC
    LIMIT 5
$$;

word_similarity is the key function. It checks whether the cleaned name appears as a substring of the member name, which handles the common case where the Hansard records only part of a member’s full name. “Kimani Ichung’wah” scores 0.52 against “Anthony Kimani Ichung’wah” via word_similarity even though similarity alone would score lower. link_speakers_by_name calls match_member for each unlinked speaker and takes the top result.

The clean_speaker_name SQL function strips honorifics, titles, constituency parentheticals, and presiding-role prefixes before matching. The Hansard produces a surprising variety of name formats:

-- Constituency and party stripped
"Hon. Kimani Ichung'wah (Kikuyu, UDA)"        → "Kimani Ichung'wah"

-- Military/professional title stripped
"Hon. (Capt.) Ronald Karauri (Kasarani, Ind)" → "Ronald Karauri"

-- Temporary Speaker with nested honorific and title
"The Temporary Speaker (Hon. (Dr) Rachael Nyamai)" → "Rachael Nyamai"

-- Hon. prefix instead of The
"Hon. Temporary Speaker (Hon. Farah Maalim)"  → "Farah Maalim"

-- Unclosed parenthesis from malformed HTML
"Hon. Kimani Ichung'wah (Kikuyu, UDA"         → "Kimani Ichung'wah"

After cleaning, trigram similarity is computed against all member names. “Kimani Ichung’wah” matches Anthony Kimani Ichung'wah at 0.52, well above the 0.45 threshold. Presiding officer patterns like “Hon. Speaker” and “The Deputy Speaker” produce false positives with fuzzy matching, so those are resolved separately via the role-based pass.

AI enrichment

Summaries are generated separately from ingestion, via the enrich subcommand of odnelazm-pipeline. The pipeline supports six enrichment targets:

All enrichment is done locally using open-source models via LM Studio. The model used to generate each summary is stored alongside it so it can be attributed in the UI and re-enriched with a better model later.

The current model is Qwen3.5 9B (qwen/qwen3.5-9b). I chose it for two reasons: its 262K native context window comfortably holds a full sitting transcript in a single call, and its built-in chain-of-thought reasoning produces noticeably more nuanced summaries than models that don’t reason before generating.

The tradeoff is inference time. At 65K context, Qwen spends the bulk of its budget on reasoning before producing the actual answer. From a recent run:

• ~14 tokens/second on Apple M-series (CPU inference)
• ~4 seconds time to first token
• ~18,000 input tokens per bill appearance summary
• ~5,000 reasoning tokens generated before the final output

At concurrency 1 on a MacBook, that works out to around 3-4 minutes per summary. A full run over 1,000 bill appearances takes several days of wall-clock time. Running on a machine with a dedicated GPU could bring this down to under 30 seconds per summary.

I will also be testing Gemma 4 e2b (google/gemma-4-e2b) as an alternative. It’s a 2B parameter model with a 128K context window, designed for on-device inference, and should be significantly faster. The quality tradeoff for complex parliamentary text is still to be evaluated.

The Summarizer trait is simple:

pub trait Summarizer: Send + Sync {
    async fn summarize(&self, prompt: &str) -> Result<String>;
}

The caller is responsible for building the full prompt. Prompt builders for each target live in enricher/prompts.rs. Bill appearance prompts include the full sitting transcript as context so the model can reference cross-bill discussions. Member contribution prompts are narrower, only including that speaker’s contributions.

Metrics

The pipeline emits metrics to a Prometheus pushgateway after each enrichment batch. Grafana is used to visualise progress, including throughput, token counts, and time to first token per call.

Running it

The pipeline is driven by a single binary, odnelazm-pipeline:

# Ingest 13th Parliament sittings
odnelazm-pipeline ingest --start-date 2022-09-01 --end-date 2026-05-19

# Enrich bill appearance summaries
odnelazm-pipeline --metrics-url http://localhost:9091 \
  enrich bill-mentions --model qwen/qwen3.5-9b --concurrency 1

Enrichment is idempotent. Items that already have a summary are skipped, so runs can be interrupted and resumed without duplicating work.

The full source is at mwananchi-tech/odnelazm. Bunge Hub is live at bunge-hub.mwananchi.tech.

Where it started: Bunge Bits

Bunge Bits started from a simple observation. The Kenyan Parliament livestreams its sittings on YouTube. Those recordings are publicly available. And yet there was no good way to search what was said, track a specific member’s contributions, or follow a bill through its debates without sitting through hours of video.

The idea was straightforward: transcribe the recordings using AI, summarize the content, and surface it in a searchable interface. I built a pipeline that pulled YouTube livestreams, ran them through a speech-to-text model, then used an LLM to generate summaries of what was discussed.

It worked, technically. The transcripts were readable, the summaries were useful, and a few people in civic and policy circles found it genuinely helpful.

Then the bills arrived.

Cloud LLMs are expensive at scale. Transcribing hours of parliamentary audio and summarizing it sitting by sitting adds up fast. The project had no funding model and no path to one. After a while, the cost of keeping it running outweighed what I could justify spending on a solo side project, and I shut it down.

The lesson I took from Bunge Bits was not that the idea was wrong. It was that I had built on the wrong foundation. AI-generated transcripts as the canonical source of truth is fragile and expensive. Parliament already publishes the official record. I just had not used it.

The pivot: Hansard as the source of truth

Kenya’s official Hansard, the verbatim record of everything said in Parliament, is published through mzalendo.com. It is structured HTML: sections, headings, contributions, speaker names. It is not perfect, but it is the authoritative record, and it is free to read.

The pivot was to stop transcribing audio and start parsing the Hansard directly. This meant building a scraper.

I wrote the first version of what would become odnelazm in Rust. The goal was simple: fetch a sitting’s HTML, parse it into structured types, and make that data accessible programmatically.

The parsing problem turned out to be more interesting than expected. mzalendo actually serves data from two distinct sources with different HTML structures: an archive at info.mzalendo.com covering sittings going back to 2006, and a current tracker at mzalendo.com/democracy-tools/hansard covering 2013 to present. Both needed separate parsers. And within the current source, the HTML structure changed around early 2026, requiring the parser to handle two formats without breaking.

That kind of structural inconsistency is normal when you are reading someone else’s website as data infrastructure. You learn to build defensively.

odnelazm: the scraper and CLI

Once the parser was solid, I wrapped it in a CLI. odnelazm-cli lets you browse and fetch parliamentary data directly from the terminal.

cargo install --git https://github.com/mwananchi-tech/odnelazm odnelazm-cli

# List recent sittings
odnelazm sittings

# Fetch a full sitting transcript (source detected automatically from the URL)
odnelazm sitting thursday-12th-february-2026-afternoon-sitting-2438

# List all members of the 13th Parliament National Assembly
odnelazm members na 13th-parliament --all -o json

# Fetch a member's full profile
odnelazm profile https://mzalendo.com/mps-performance/national-assembly/13th-parliament/boss-gladys-jepkosgei/

It supports both archive and current data sources with automatic routing, outputs JSON, CSV, or Parquet, and handles pagination. It is the kind of tool I wished existed when I was building Bunge Bits.

odnelazm-mcp: asking questions about Parliament in plain English

The next step was making the data accessible to LLM clients. I built odnelazm-mcp, an MCP (Model Context Protocol) server that exposes the scraper as tools, so any compatible client (Claude or ChatGPT) can query parliamentary data directly in conversation.

cargo install --git https://github.com/mwananchi-tech/odnelazm odnelazm-mcp-local

With it connected to Claude Desktop, you can ask things like:

• “Who spoke on the Finance Bill in June 2023?”
• “List all Senate sittings from the first quarter of 2025”
• “What did Gladys Boss Shollei say about housing?”

The server handles the fetching and parsing transparently. The LLM reasons over the structured data.

The limitation is context. Hansard transcripts are large. A single sitting can run to tens of thousands of tokens. Fetching multiple sittings, or a member’s full activity history, can easily exceed the context window of most models. The practical guidance is to keep queries narrow: one sitting at a time, one date range at a time. For broad cross-sitting queries, a local model with a large context window (1M+ tokens) handles it better.

Bunge Hub: the full picture

odnelazm as a scraper and CLI is useful for developers and researchers who are comfortable in a terminal. But most people who care about parliamentary accountability are not that audience.

Bunge Hub is the web layer on top of the same data. It runs a PostgreSQL database populated by odnelazm-pipeline, which handles ingesting sittings, extracting bills and topics, importing member profiles, and linking speaker records to canonical member identities.

The data model that emerged from this is something I am genuinely proud of. 885 sittings, 391 bills, 9,826 topics, 3,001 speaker records linked where possible to 429 members. You can trace a bill through every sitting it appeared in, see who spoke at each stage, and read their contributions side by side.

The linking problem, matching a speaker name extracted from a transcript to a canonical member record, turned out to be one of the harder engineering problems in the project. The same person appears as “Hon. Kaluma”, “George Kaluma”, “Peter Kaluma” across different sittings. The solution combines exact URL matching (when mzalendo includes a profile link), fuzzy trigram similarity via pg_trgm, and a handful of special cases, like the Speaker of the National Assembly, whose contributions are labelled “Hon. Speaker” and do not fuzzy-match anything useful.

On top of this, I added an AI enrichment layer using locally running open-source models via LM Studio. Each bill appearance gets a summary of what was argued and what the outcome was. Each bill’s full journey across Parliament gets a narrative summary. The model used to generate each summary is recorded and displayed in the UI.

Doing it sustainably this time

The core lesson from Bunge Bits was about cost. I am not repeating that mistake.

The enrichment pipeline runs locally. No cloud LLM APIs, no per-token billing, no dependency on a service that can change its pricing or shut down. The tradeoff is throughput: a 9B model on consumer hardware is slow. But the data does not expire. A summary generated this week is just as useful next month.

The architecture is designed so that if better hardware or funding becomes available, swapping to a faster model is a one-line change. Every summary is attributed to the model that generated it, so a future re-enrichment pass can target specific rows selectively.

There is also no funding dependency for the core product. The database and web app cost pennies to host at this scale. The ingest pipeline runs on a cron job. If I step away for six months, the data already in the database stays useful.

One other deliberate decision: I created Mwananchi Tech as an open-source organisation to be the umbrella for these projects, rather than keeping them under my personal GitHub. The practical reason is that associating the work with an organisation rather than an individual makes it easier to attract collaborators and potential partners. If someone wants to contribute or build on top of what is here, they are joining a project, not someone’s personal experiment.

What’s next

Bunge Hub is in its later stages of development. The code is open source at mwananchi-tech/bunge-hub, and a full launch post will be published once it is publicly available.

The current data covers the 13th Parliament (September 2022 to present). The archive goes back to 2006. odnelazm already has a full parser for the archival source at info.mzalendo.com, so extending coverage to earlier parliaments is a matter of running the pipeline, not building new infrastructure.

The other thing I keep thinking about is YouTube. The recordings that Bunge Bits used to transcribe are still there. Now that the Hansard is the canonical source, the opportunity is different: not transcription, but timestamping. If you could link a specific contribution in the structured transcript to the exact moment in the YouTube recording, you could read a debate and click directly to the video. That closes the loop between the written record and the live moment in a way nothing else currently does.

That is a harder problem. But the foundation is now solid enough to build it on.

Why am I doing this?

Honestly, curiosity got me here. There is a genuine gap nobody in the Kenyan tech space has seriously tried to close, and that bothered me enough to do something about it.

I did worry, at some point, about tying my identity too closely to civic tech. It is a niche space and not always the most visible one. But I have stopped caring about that.

There was also a more uncomfortable fear. People who work in civic tech in Kenya get reminded, sometimes only half-jokingly, that Subarus might come for them. That fear is not entirely abstract. What happened to Rose Njeri was a real reminder of the risks that come with this kind of work in this environment. I thought about it. I will not pretend I did not. But I am confident that what I am building is not illegal. All the data I use is publicly available. The Hansard is the official parliamentary record. I am not hacking anything, exposing anything that is not already public, or doing anything I could not defend clearly and loudly if I had to.

A few days ago I attended a Tech and Democracy meetup organized by Sparkable and met some genuinely interesting people working on accountability and governance from different angles. That conversation spurred me on more than anything else has in a while. If this is the cause my name ends up attached to, I am fine with it. It is technically challenging, it is for the greater good, and it is using the latest tooling and AI to bring some meaning to an era where that can be hard to find.

Also, full transparency: I have had a lot of free time. Not working a full-time job at the moment, which is partly how a project this scope got built by one person in under a year. If you are hiring and any of this resonates, I am open to it.

This project was an excuse to go deeper.

The Library of Babel

The Library of Babel is a short story by Jorge Luis Borges. The premise is a library that contains every possible 410-page book. Not just the books that have been written, but every combination of characters that could exist. Your biography, before you have lived it. This documentation. Complete nonsense. Every text, in every order, somewhere on a shelf.

Jonathan Basile built a working web implementation of this. A few months ago, this video by Merlin on YouTube was randomly recommended to me. I watched it and immediately fell into a rabbit hole about deterministic randomness.

The interesting part is not that the library contains everything. It is how it does it. The content of any page is not stored anywhere. It is computed on demand, deterministically, from its address. Search for a piece of text, and you get back a location in the library where that exact text appears, generated from the search itself as a seed. The library does not remember your query. It just consistently gives you the same page for the same input every time.

That idea stuck with me. I wanted to build it.

The Character Set and Base29 Encoding

The original Borges concept uses a small alphabet. Most implementations converge on a 29-character set: lowercase a to z, a space, a comma, and a period. That gives you enough to write something coherent, and 29 is a prime number, which has nice properties for base conversion. Some implementations go further: this TypeScript implementation by tdjsnelling extends the charset to 32 characters. I chose to stick with the original 29. I did borrow the addressing system from that project though, which I found quite intuitive.

charset := " abcdefghijklmnopqrstuvwxyz,."

Each page in the library is 3,200 characters (40 lines of 80 characters). To find a page, you convert its content into a single base-29 number. That number, decomposed into its hierarchical parts, gives you the address. The conversion treats each character as a digit in base 29: space = 0, a = 1, b = 2, and so on through to period = 28.

Going the other way (address back to page content) is repeated division by 29, collecting remainders, then mapping each remainder back to a character. The library is just a very large number space dressed up as rooms and shelves.

Seeding the Search

The thing that makes search work is using the search text itself as a seed to generate a deterministic position. When you search for “hello”, the library does not scan billions of pages looking for the word. It hashes “hello” with SHA256, seeds an RNG from that hash, generates a full 3,200-character page with “hello” planted at a position within it, then converts that page to its base-29 address.

func (l Library) seedPageChars(text string, variant int) string {
    input := fmt.Sprintf("%s\x00%d", strings.ToLower(text), variant)
    textHash := sha256.Sum256([]byte(input))
    textSeed := int64(binary.BigEndian.Uint64(textHash[:8]))
    rng := rand.New(rand.NewSource(textSeed))

    maxPosition := charsPerPage - len(text)
    position := rng.Intn(maxPosition + 1)

    pageChars := make([]byte, charsPerPage)
    for i := range charsPerPage {
        pageChars[i] = l.charset[rng.Intn(len(l.charset))]
    }

    copy(pageChars[position:], []byte(strings.ToLower(text)))
    return string(pageChars)
}

seedPageChars is called by generateBase29Number, which walks the page character by character and builds the actual base-29 number:

func (l Library) generateBase29Number(text string, variant int) (*big.Int, error) {
    result := big.NewInt(0)
    pageChars := l.seedPageChars(text, variant)

    for _, char := range pageChars {
        result.Mul(result, l.base)
        index := l.charToIndex[char]
        result.Add(result, big.NewInt(int64(index)))
    }

    return result, nil
}

Each character is treated as a digit: multiply the running total by 29, add the character’s index. After 3,200 characters you have a single very large integer that uniquely encodes the page. That number is what gets decomposed into the hexagon, wall, shelf, book, and page coordinates.

The variant parameter is how multiple results work. Searching with variant 0 gives you one address, variant 1 gives you a different one. The same text plus a different variant integer produces a different seed, a different page, a different address. Every text “appears” in the library an essentially unlimited number of times, just at different coordinates.

How Many Results?

This is where the math got interesting, and I will be honest: I relied on AI a lot here because combinatorics is not really my forte.

The number of locations where a given text appears should scale inversely with the length of that text. A single character like “a” appears everywhere. The phrase “to be or not to be” appears much less frequently. A full paragraph is vanishingly rare.

I modeled this with exponential decay:

const exponentialDecayRate = 1.10

func (l Library) GetOccurrenceCount(text string) int {
    textLen := len(text)
    maxCount := 1_000_000_000 // 1 billion for single characters
    baseCount := maxCount / int(math.Pow(exponentialDecayRate, float64(textLen-1)))
    baseCount = max(1, baseCount)

    hash := sha256.Sum256([]byte(strings.ToLower(text)))
    seed := int64(binary.BigEndian.Uint64(hash[:8]))
    rng := rand.New(rand.NewSource(seed))

    variation := max(1, baseCount/4)
    adjustment := rng.Intn(2*variation) - variation

    return max(1, baseCount+adjustment)
}

Starting at 1 billion for a single character, each additional character divides the result by 1.10. By the time you reach a moderately long phrase, you are down to a handful of results. The ±25% variation seeded from the text hash gives each query a consistent but unique feel. The tradeoff is that the billion cap is completely hardcoded. It is a reasonable approximation of “effectively infinite”, not something derived from first principles. Good enough for this.

The Addressing System

Every page in the library has a five-part address: <hexagon>.<wall>.<shelf>.<book>.<page>. The structure mirrors Borges’ description: hexagonal rooms with four walls, each holding five shelves, each shelf holding 32 books, each book with 410 pages.

Converting a base-29 number to an address is a series of modular divisions:

func newFromBase29Number(n *big.Int) *Location {
    temp, quotient := new(big.Int).Abs(n), new(big.Int)

    page := new(big.Int)
    quotient, page = quotient.DivMod(temp, big.NewInt(pagesPerBook), page)
    temp.Set(quotient)

    book := new(big.Int)
    quotient, book = quotient.DivMod(temp, big.NewInt(booksPerShelf), book)
    temp.Set(quotient)

    shelf := new(big.Int)
    quotient, shelf = quotient.DivMod(temp, big.NewInt(shelvesPerWall), shelf)
    temp.Set(quotient)

    wall := new(big.Int)
    quotient, wall = quotient.DivMod(temp, big.NewInt(wallsPerHexagon), wall)
    temp.Set(quotient)

    return &Location{
        Hexagon: quotient.Text(36),
        Wall:    int(wall.Int64()),
        Shelf:   int(shelf.Int64()),
        Book:    int(book.Int64()),
        Page:    int(page.Int64()) + 1,
    }
}

The hexagon identifier is whatever is left after stripping out wall, shelf, book, and page. It can be an astronomically large number, so it is stored as a base-36 string (digits and lowercase letters). An address ends up looking something like 3q7z.2.1.14.207. The hierarchy is reversible: you can reconstruct the original base-29 number from any valid address, which is how browsing works.

Pagination

One thing the original concept glosses over is what pagination even means when a search for “a” returns close to a billion results.

The variant approach makes this tractable. Variants are sequential integers starting from 0. Pagination is just offset and limit over that sequence:

func (l Library) SearchPaginated(text string, offset, limit int) ([]*Location, error) {
    totalCount := l.GetOccurrenceCount(text)

    if offset < 0 {
        return nil, errors.New("offset cannot be negative")
    }
    if limit <= 0 {
        return nil, errors.New("limit must be positive")
    }
    if offset >= totalCount {
        return []*Location{}, nil
    }

    endIndex := min(offset+limit, totalCount)
    locations := make([]*Location, 0, endIndex-offset)

    for variant := offset; variant < endIndex; variant++ {
        bigInt, err := l.generateBase29Number(text, variant)
        if err != nil {
            return nil, fmt.Errorf("error generating location for variant %d: %w", variant, err)
        }
        locations = append(locations, newFromBase29Number(bigInt))
    }

    return locations, nil
}

Each page of results is a slice of variant integers. No state, no cursors, no database. You can reproduce any page of results for any query deterministically.

Hacks and Loose Ends

There are a few places in the code where I left honest markers of imperfection.

The significant one is in base29NumberToString. The browse API accepts any valid address, and hexagon identifiers can be any integer from 1 up to the maximum big.Int. If you manually navigate to a small address like 1.0.0.0.1, the corresponding base-29 number is tiny, and decoding it back to characters produces a single-character page. A page with one character is not ideal. The hack is to take whatever characters the conversion produced, hash them, seed an RNG from that hash, and fill the remaining space. It preserves determinism (the same small address always fills the same way) but it is not elegant.

Concurrency and the Worker Pool

SearchStream was my first real exercise in Go concurrency. Rather than blocking until all results are computed, it returns a channel immediately and lets the caller consume locations as they are generated. The full implementation looks like this:

func (l Library) SearchStream(ctx context.Context, text string) (<-chan *Location, error) {
    totalCount := l.GetOccurrenceCount(text)
    locationChan, workerChan := make(chan *Location), make(chan int, 100)
    numWorkers := runtime.NumCPU()
    var wg sync.WaitGroup

    for range numWorkers {
        wg.Go(func() {
            for {
                select {
                case <-ctx.Done():
                    return
                case variant, ok := <-workerChan:
                    if !ok {
                        workerChan = nil
                        continue
                    }
                    bigInt, err := l.generateBase29Number(text, variant)
                    if err != nil {
                        continue
                    }
                    location := newFromBase29Number(bigInt)
                    select {
                    case locationChan <- location:
                    case <-ctx.Done():
                        return
                    }
                }
            }
        })
    }

    go func() {
        defer close(workerChan)
        for variant := range totalCount {
            select {
            case workerChan <- variant:
            case <-ctx.Done():
                return
            }
        }
    }()

    go func() {
        defer close(locationChan)
        wg.Wait()
    }()

    return locationChan, nil
}

There are three moving parts.

The dispatcher goroutine sends variant integers (0, 1, 2…) into a buffered workerChan. The buffer size of 100 means the dispatcher can stay ahead of the workers without blocking. It stops early if the context is cancelled.

The worker goroutines (one per CPU) pull variants from workerChan, compute a location, and send it to locationChan. Each of the two send operations checks ctx.Done() so that a cancelled context unblocks workers whether they are waiting to receive a job or waiting to send a result.

The closer goroutine waits for all workers to finish via sync.WaitGroup, then closes locationChan. The consumer knows the stream is exhausted when the channel closes.

Goroutine Leaks

The more interesting case is when you only want a subset of results. The test below reads exactly 100 locations then stops:

func TestLibrarySearchStream(t *testing.T) {
    library := New()
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    results, err := library.SearchStream(ctx, searchText)
    if err != nil {
        t.Errorf("search stream failed: %v", err)
    }

    locations := []*Location{}
    limit := 100
    for range limit {
        location := <-results
        locations = append(locations, location)
    }
    cancel()

    if l := len(locations); l != limit {
        t.Errorf("expected %d locations, got %d", limit, l)
    }
}

Once the loop exits, we call cancel() explicitly rather than relying on defer. Both work, but calling it immediately after collecting the results we need is the right instinct: it signals the workers to stop as soon as possible rather than waiting until the function returns.

Without that cancellation, you get a goroutine leak. Here is the scenario. locationChan is unbuffered. Once the consumer stops reading, the next worker trying to send a location will block forever on this:

select {
case locationChan <- location: // blocked: nobody reading
case <-ctx.Done():             // never fires: context not cancelled
    return
}

All workers stall. wg.Wait() never returns. locationChan is never closed. Everything is stuck.

Cancelling the context is what breaks the deadlock. The case <-ctx.Done() branch fires in every worker, they all return, wg.Wait() completes, and locationChan closes. Whether you call cancel() explicitly after collecting what you need or rely on defer, the important thing is that it always runs.

The dispatcher has the same guard. If the context is cancelled while the dispatcher is still enqueuing variants, it exits immediately rather than blocking on a full workerChan:

select {
case workerChan <- variant:
case <-ctx.Done():
    return
}

The Nil Channel Trick

There is one subtlety in how workers handle a closed workerChan. When the dispatcher has sent all variants and its goroutine exits, it closes workerChan. A worker receives the zero value with ok = false, then does something that looks odd:

case variant, ok := <-workerChan:
    if !ok {
        workerChan = nil
        continue
    }

It sets its local copy of workerChan to nil and loops. A receive on a nil channel blocks forever. On the next iteration, the worker’s select effectively becomes:

select {
case <-ctx.Done():
    return
case variant, ok := <-nil: // blocks forever, never selected
    ...
}

The worker is now parked, waiting only for context cancellation. This is intentional. The alternative would be for the worker to return immediately when workerChan closes, but that would decrement the WaitGroup before the worker has necessarily finished sending its last computed location to locationChan. Staying alive until ctx.Done() ensures the worker fully drains before the WaitGroup completes and locationChan is closed.

Coming from Rust, this all felt foreign. In Rust the compiler enforces the invariants: ownership, Send/Sync bounds, lifetime constraints. In Go you reason about it yourself and encode the contract in patterns like this one. The channel syntax took time to internalize, especially select across multiple channels. I leaned heavily on Learning Go by Jon Bodner throughout this project, particularly for the concurrency chapter and general idiomatic patterns. It is a well-written book and I would recommend it to anyone coming from a statically typed language who finds Go’s conventions a bit opaque at first. The mental model holds up once it clicks. Channels are typed queues, goroutines are lightweight threads, and the patterns (worker pool, fan-out, context cancellation) are consistent everywhere you look in the ecosystem.

The CLI and Web App

You can interact with the library through a CLI or the web app at babel.c12i.xyz. For the CLI, I used kong. My bias towards it is straightforward: coming from Rust, I am used to defining CLI commands as structs and letting the framework derive the parser from struct field tags, which is exactly how clap works. kong is the closest equivalent in Go, and that familiarity made it an easy call over something like cobra.

For the web layer, I used Gin. It is fast, the routing API is clean, and the choice did not need much deliberation. The UI is mostly AI-generated. I gave Claude some rough requirements for a search form, results list, and browse view with previous/next navigation, and iterated from there. The interesting parts of this project were in the library core, not the frontend.

This was a genuinely enjoyable side project. It pushed me further into Go than IaC work ever did, gave me a working understanding of goroutines and channels, and produced something I find kind of beautiful: an infinite library you can search, that requires no storage, generates every page on demand, and always gives you the same answer for the same question.

The code is open source at github.com/c12i/babel-go. The web app is live at babel.c12i.xyz. Go check what the library has to say about your name.

I built bunge-bits out of frustration. Kenya’s National Assembly and Senate regularly livestream their proceedings, but these sessions are long, often poorly indexed, and inaccessible to most people. The average citizen doesn’t have the time, or bandwidth, to sit through up to five-hour videos of parliamentary debates. Most people end up relying on the media to highlight key moments, often missing the full context or nuance.

At the same time, Kenya is experiencing a generational shift. Young people are increasingly vocal about governance, public spending, and corruption. I wanted to contribute something tangible to that movement, something that makes civic discussions more transparent, digestible, and shareable.

Bunge Bits is my attempt to bridge that gap: an AI-powered pipeline that listens to Parliament so citizens don’t have to, and summarizes the highlights in plain markdown.

The Stack

This project leans heavily on modern AI tools and a reliable Rust backbone:

• Rust: For performance, error handling, and concurrency.
• yt-dlp: To fetch livestream recordings from YouTube.
• ffmpeg: For slicing audio into manageable chunks.
• OpenAI Whisper: To transcribe spoken audio to text.
• GPT-4: To summarize transcripts into clear, structured markdown.
• Postgres: For storing metadata and summaries.
• Cron jobs: For scheduling runs and cleaning up old artifacts.

Here’s how data flows through the system:

Engineering Challenges

Some parts of this were painful:

• Whisper often mangled Kenyan names: “John Mbadi” became “John Birdy”, and it only got worse with native pronunciations.
• Bad transcriptions created hallucinated summaries: GPT-4 would fill in gaps with confident nonsense.
• OpenAI context limits: Long transcripts could easily exceed GPT’s token window, making summarization error-prone or impossible.
• Multiple streams per day: My VPS filled up fast with raw audio and transcripts.
• Chunk order and retries: Ensuring chunks were in sequence and retried correctly took careful design.
• Summary formatting: I needed markdown output with clean headings, bullets, and time-stamped key moments.

Solutions and Architecture Decisions

To tackle these, I made a few key calls:

• Name mangling and hallucinations: To improve recognition of Kenyan public figures, I enrich GPT-4 prompts with curated lists of MPs, Senators, and Cabinet Secretaries. This leverages both GPT’s internal knowledge and its ability to ground responses via web search, reducing hallucinations and transcription errors.
• System/user prompt separation: This helps enforce markdown formatting, avoid emojis, and ensure consistent structure.
• Linear, contextual summarization pipeline: This was the biggest challenge. To avoid exceeding OpenAI’s context window, I chunk transcripts and summarize each piece separately, feeding each new summary back in as context for the next. The final output is then stitched together in a “combine” step. This closure-based approach let me scale summaries without blowing the token limit.
• Batching via cron: The job runs every 12 hours, fetching and processing only new streams. To conserve storage on the VPS, each run processes a fixed number of unprocessed streams (currently capped at 3), prioritized by upload date. This prevents overload during periods of high parliamentary activity.
• Separate cron for cleanup: A lightweight system cron wipes old audio/transcripts after processing to avoid disk bloat.

What’s Next

There’s a lot more I want to do (in order of priority):

• A public UI where users can browse summaries and filter by topic or MP.
• Automatic timestamp injection for key moments (e.g. heated debates, bill passages, speaker interventions) to make summaries easier to navigate — this is a planned feature.
• The repo is already open source, but I plan to apply for OpenAI credits, civic tech grants, or journalism fellowships to fund ongoing compute and infrastructure costs.
• Eventually, I’d love to collaborate with civil society groups, media houses, or transparency orgs to expand its reach.

View the code on GitHub

Welcome to Rust, where safety and zero-cost abstractions reign supreme, but sometimes the simplest tasks can become unexpected challenges.

The Problem: Rust’s Compile-Time Conundrum

Why can’t Rust handle simple string concatenation at compile time? Well, it’s not as straightforward as you might think.

Let’s look at an example that seems like it should work, but doesn’t:

use clap::Parser;

const TOOL_NAME: &str = "SuperTool";
const COMPATIBLE_CRATE: &str = "AwesomeDependency";
const COMPATIBLE_VERSION: &str = "1.2.3";

// This doesn't work!
const ABOUT: String = format!(
    "{} - Compatible with {} version {}",
    TOOL_NAME, COMPATIBLE_CRATE, COMPATIBLE_VERSION
);

#[derive(Parser, Debug)]
#[command(author = "Alice", version, about = ABOUT)]
struct Cli {
    // CLI arguments would go here
}

fn main() {
    let args = Cli::parse();
    println!("{:?}", args);
}

Try to compile this, and Rust responds with an error:

error[E0015]: calls in constants are limited to constant functions, tuple structs and tuple variants
 --> src/main.rs:8:21
  |
8 | const ABOUT: String = format!(
  |                       ^^^^^^ ...

error: aborting due to previous error

The format! macro, usually so helpful for string formatting, suddenly becomes unavailable in the compile-time context.

Rust’s const evaluation capabilities, while continuously improving, still have limitations. The language designers have prioritized other features, leaving compile-time string formatting as a challenge to overcome.

The Solution: Leveraging Crates

Fortunately, the Rust ecosystem provides solutions. One such solution is the const_format crate, which allows for compile-time string formatting.

With const_format, you can write:

use const_format::formatcp;

const TOOL_NAME: &str = "SuperTool";
const COMPATIBLE_CRATE: &str = "AwesomeDependency";
const COMPATIBLE_VERSION: &str = "1.2.3";
const ABOUT: &str = formatcp!(
    "{} - Compatible with {} version {}",
    TOOL_NAME, COMPATIBLE_CRATE, COMPATIBLE_VERSION
);

This approach gives us the compile-time string formatting we need.

An Alternative: Build Scripts

There’s another way to achieve compile-time string formatting without additional crates: build scripts and environment variables.

By using a build.rs file, we can generate our formatted strings at compile time and pass them to our main code as environment variables. Here’s how:

1. Create a build.rs file in your project root:

fn main() {
    let tool_name = "SuperTool";
    let compatible_crate = "AwesomeDependency";
    let compatible_version = "1.2.3";
    let about = format!(
        "{} - Compatible with {} version {}",
        tool_name, compatible_crate, compatible_version
    );

    println!("cargo:rustc-env=ABOUT={}", about);
}

1. In your main.rs, use the environment variable:

use clap::Parser;

#[derive(Parser, Debug)]
#[command(author = "Alice", version, about = env!("ABOUT"))]
struct Cli {
    // CLI arguments would go here
}

fn main() {
    let args = Cli::parse();
    println!("{:?}", args);
}

This method allows us to perform the string formatting during the build process and inject the result into our main code.

Lessons Learned

This journey through compile-time string formatting in Rust teaches us a few valuable lessons:

1. Rust prioritizes safety and performance, sometimes at the cost of convenience.
2. The Rust ecosystem often provides solutions for language limitations.
3. Build scripts are powerful tools that can help overcome certain restrictions.
4. Sometimes, the simplest solutions (like using environment variables) can solve complex problems.
5. With some creativity, we can usually find a way around language limitations.

While Rust may not support compile-time formatting out of the box, we have multiple ways to achieve our goals. Whether it’s through community-created crates or clever use of build scripts, Rust developers always find a way forward.

Introduction

In this tutorial, we will explore the fascinating intersection of Rust, Web Assembly (Wasm), and React by building a simple password generator. We’ll leverage the speed and performance benefits of Rust, compile it into Wasm, and seamlessly integrate it into a React application. By the end of this project, you’ll have a clear understanding of how these technologies work together.

Why Rust and Web Assembly?

Web Assembly is a low-level virtual machine that allows executing code at near-native speeds within web browsers. Rust, known for its speed and memory safety, is an ideal language for Wasm. Unlike high-level languages like JavaScript, Rust provides developers with low-level control and reliable performance, making it an excellent choice for Wasm.

Prerequisites

Before we dive into the project, ensure you have the following tools and knowledge:

• Rust and Cargo: Install them using this link.
• wasm-pack: A tool for building and working with Rust-generated Web Assembly. Install it here.
• Node.js: Required for React development. Install it from nodejs.org.
• Basic knowledge of Rust and JavaScript/React.js.

Initializing the Rust Project

Let’s start by creating a new Rust Web Assembly project using the following command:

wasm-pack new wasm_pass

This command generates a Rust library crate with Web Assembly support. The crucial files are Cargo.toml, src/lib.rs, and src/utils.rs.

Dissecting the Boilerplate

Explore the generated project structure:

wasm_pass/
├── Cargo.toml
├── LICENSE_APACHE
├── LICENSE_MIT
├── README.md
└── src
    ├── lib.rs
    └── utils.rs

• Cargo.toml: Contains project dependencies and metadata.
• lib.rs: The root of the crate, compiled into Web Assembly.
• utils.rs: Includes debugging utilities (not used in this project).

Initial Build

Build the project using:

wasm-pack build

This creates a pkg directory with the following artifacts:

pkg/
├── package.json
├── README.md
├── wasm_pass.wasm
├── wasm_pass.d.ts
└── wasm_pass.js

These files enable npm integration and contain the compiled Web Assembly code.

Writing Rust Code

Update the Cargo.toml file to include the rand crate:

[dependencies]
wasm-bindgen = "0.2.63"
rand = { version = "0.7.3", features = ["wasm-bindgen"] }

This adds the random number generation utility to our project. Run cargo build to download the new dependency.

Implement the password generator logic in src/lib.rs:

use wasm_bindgen::prelude::*;
use rand::Rng;

#[wasm_bindgen]
pub fn generate(len: usize) -> String {
    const CHARSET: &[u8] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ\
                            abcdefghijklmnopqrstuvwxyz\
                            0123456789)(*&^%$#@!~".as_bytes();

    let mut rng = rand::thread_rng();

    let password: String = (0..len)
        .map(|_| {
            let idx = rng.gen_range(0, CHARSET.len());
            CHARSET[idx] as char
        })
        .collect();
    password
}

This function generates a random password of the specified length using the rand crate.

Testing the Code

Write a test for the generate function in src/lib.rs:

#[cfg(test)]
mod tests {
    use super::generate;

    #[test]
    fn test_generate() {
        let password = generate(20);
        assert_eq!(password.len(), 20);
    }
}

Run the tests with:

cargo test -- --show-output

Ensure the tests pass, confirming the functionality of the password generator.

Implementing Web Assembly in a React App

Create a React app in the app directory:

mkdir app
cd app
mkdir src
echo "/node_modules" > .gitignore
npm init -y

Install necessary dependencies:

npm install --save react react-dom
npm install --save-dev @babel/core @babel/preset-env @babel/preset-react babel-loader webpack webpack-cli webpack-dev-server html-webpack-plugin style-loader css-loader html-loader

Create the React app files: index.html, index.js, and App.js. Configure Babel with .babelrc and Webpack with webpack.config.js.

Finally, run the development server:

npm start

Using the Package in React

Update the App.js file to use the Rust-generated Web Assembly package:

import React, { useState, useCallback, useEffect } from "react";

const App = () => {
    const [password, setPassword] = useState("");
    const [input, setInput] = useState("");

    const handleChange = (e) => {
        setInput(e.target.value);
    };

    const generatePassword = useCallback(() => {
        const module = import("wasm-pass")
        module.then(({generate}) => {
            setPassword(generate(parseInt(input)))
        }).catch(err => {
            alert(err.toString())
        })
    }, [input]);

    useEffect(() => {
        generatePassword()
    }, [input]);

    return (
        <div>
            <p>Enter password length:</p>
            <input onChange={handleChange} type="number" value={input} />
            <button onClick={generatePassword}>Generate Password</button>
            <p>Your password:</p>
            <strong>{password}</strong>
        </div>
    );
};

export default App;

This React component allows users to input a password length, click a button, and receive a randomly generated password from the Rust Web Assembly package.

Conclusion

This tutorial showcases the integration of Rust and Web Assembly in a React application, providing a powerful blend of performance and modern front-end technologies. Explore further resources to deepen your understanding of Rust and Web Assembly:

• The Rust Book
• Rust and Web Assembly Book
• wasm-bindgen Documentation
• Wasm By Example
• WebAssembly.org

Check the full working implementation of this project for reference. Happy coding!

If you’re frequently developing and deploying Node.js applications, encountering the infamous FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed error is not uncommon. This issue arises particularly when your server engages in memory-intensive tasks, such as handling large data transfers with limited system resources.

Understanding the V8 Engine

Node.js relies on Google’s V8 engine, an open-source, high-performance JavaScript and WebAssembly engine written in C++. Integrated into both Chrome and Node.js, V8 implements ECMAScript and WebAssembly and caters to various operating systems and processor architectures.

When a Node.js process begins, V8 allocates a specific amount of memory based on your hardware specifications. Monitoring and managing this allocated memory is crucial, especially when dealing with resource-demanding applications.

Checking Allocated Memory

To gain insights into your process’s allocated memory, create an index.js script with the following code:

const v8 = require("v8");

const totalHeapSize = v8.getHeapStatistics().total_available_size;
let totalHeapSizeInGB = (totalHeapSize / 1024 / 1024 / 1024).toFixed(2);
console.log(
    `Total Heap Size is ${totalHeapSize} bytes`,
    `${totalHeapSizeInGB} GB`
);

This script utilizes the v8 module, an integral part of Node.js, offering methods like v8.getHeapStatistics() to fetch heap-related memory information.

Adjusting Memory Capacity

You can increase the memory capacity by using the --max-old-space-size argument when running your script, specifying the desired memory allocation in kilobytes:

node --max-old-space-size=4096 index.js

Upon rerunning the script, you should observe an augmented total heap size:

Total Heap Size is 4283117000,   GB 3.99

In this example, the memory has been doubled, providing more breathing room for your processes.

Important Considerations

Before adjusting memory allocation, ensure you have sufficient free memory. Attempting to allocate more memory than available on your system is counterproductive.

Moreover, prioritize refactoring your code for better memory efficiency as a primary strategy before resorting to memory allocation adjustments.

Conclusion

Optimizing memory allocation is a key aspect of maintaining high-performance Node.js applications. By understanding the V8 engine’s role and implementing judicious memory adjustments, you can enhance your application’s stability and responsiveness. Always balance these optimizations with efficient code practices to achieve optimal results.