Worker Architecture Specification
The stateful worker layer of the Ashlight Focus Engine. Where the core engine handles search, display, and collection, Rabbit Runner handles execution — the jobs that run in the background, persist state, and report back to Director.
DRAFT — Architectural Earmark
Ashlight Systems Inc. | Shane Tyas | v0.1 | 24 Feb 2026
Read Architecture Director Commands
01
What Is Rabbit Runner
A Rabbit is a named worker instance. Each rabbit has a type, a target, a strategy, and a job queue. Rabbits are not hardcoded — they are spawned from a JSON index, take commands from Director via the command bus, and can be stopped, resumed, and inspected at any time.
The name reflects the behaviour: Rabbits go down holes (into data sources), follow tunnels (link graphs, TOC trees, file hierarchies), surface with results, and report to the warren (Director).
02
Why a Separate Worker Layer
The current crawl implementation is synchronous, single-strategy, and stateless. It works but has no memory between runs, no domain awareness, and no ability to run parallel jobs.
Azure/Edge Noise
Pages from Microsoft Learn that have nothing to do with Bedrock — navigation chrome, Edge download prompts, privacy pages. The crawler follows all links on the same hostname, so it ingests everything.
Blind Link-Following
No awareness of document hierarchy. The Bedrock docs have a toc.json that describes the full tree — the crawler ignores it and tries to discover pages by following anchor tags.
No Filter Persistence
Noise filters need to be re-applied every crawl. There is no persistent record of which URL patterns are known noise for which domain.
Single Strategy
Microsoft Learn, GitHub repos, and Stack Overflow all need different traversal strategies. The current crawler applies the same link-follow approach to all proxy repos.
No Job Memory
If a crawl is interrupted, it restarts from zero. No checkpoint, no resume.
03
Architecture Overview
Rabbit Runner sits between the core engine and the data sources. Director commands it. The core engine receives its output.
Director
→
Rabbit Runner
→
Data Sources
Core Engine
←
Results
←
Rabbit Runner
Component Description
repo.index.json The source of truth for all known repos. Contains domain, strategy type, known TOC endpoints, noise filter patterns, seed URLs, and metadata. Rabbit Runner reads this before spawning any worker.
Rabbit (worker) A single worker instance. Has a type (toc | crawl | github | sitemap), a target repo ID, a job queue (array of URLs/paths to fetch), a visited set, and a results buffer.
Job queue Ordered array of work units. Can be pre-populated from toc.json or grown dynamically by link-following. Queue is serialisable — can be saved and resumed.
Filter index Per-domain array of URL pattern strings that indicate noise. Stored in repo.index.json. Applied before any URL enters the job queue.
Director commands spawn rabbit, kill rabbit, status rabbits, pause rabbit, resume rabbit, add filter, show index
Emit events RABBIT_START, RABBIT_PROGRESS, RABBIT_COMPLETE, RABBIT_ERROR, RABBIT_PAUSED. Same event bus as current HANDSHAKE/COMPLETE.
04
Strategy Types
Each repo in the index specifies a traversal strategy. The strategy determines which worker type is spawned and how it executes.
toc
Fetch the toc.json endpoint first. Parse it into a flat URL list. Use that as the job queue. No link-following needed — the TOC is the scenegraph.
crawl
Classic link-following. Seeds the queue from baseUrl or provided seeds, follows links on the same domain, applies noise filters. Used when no TOC or API is available.
github
Uses GitHub Contents API to get the full file tree in one request. Filter by extension or path prefix. Fetch individual files directly from raw.githubusercontent.com.
sitemap
Fetches sitemap.xml or robots.txt first, parses URL list, queues targeted fetches. Faster than crawl when a sitemap exists.
api
Reserved for structured data endpoints (REST, GraphQL). Knows how to paginate and extract. Not yet implemented.
05
repo.index.json — The Strategy Index
This file is the semantic knowledge base for traversal. It is not hardcoded in any JavaScript. It lives in the project root and is read by Rabbit Runner at spawn time. 
{
  "bedrock_docs": {
    "id": "bedrock_docs",
    "label": "Bedrock Creator Docs",
    "domain": "learn.microsoft.com",
    "baseUrl": "https://learn.microsoft.com/en-us/minecraft/creator/",
    "strategy": "toc",
    "toc": "https://learn.microsoft.com/en-us/minecraft/creator/toc.json",
    "seeds": [
      "https://learn.microsoft.com/en-us/minecraft/creator/reference/content/entityreference/",
      "https://learn.microsoft.com/en-us/minecraft/creator/reference/content/blockreference/"
    ],
    "noiseFilters": [
      "/azure/", "/edge/", "/windows/", "/visualstudio/",
      "download-microsoft-edge", "privacy", "terms-of-use",
      "lifecycle-faq", "entra-id", "subscription"
    ],
    "meta": {
      "addedBy": "Director",
      "addedAt": "2026-02-24",
      "crawlCount": 0,
      "lastCrawl": null
    }
  }
}
06
Noise Filter System
Noise filters are URL pattern strings stored per domain in repo.index.json. Before any URL is added to a rabbit's job queue, it is tested against the filter list.
Filters are not hardcoded. They are built up over time as Director identifies noise patterns. Director command:

add filter learn.microsoft.com /azure/

This appends "/azure/" to the noiseFilters array for the learn.microsoft.com domain in repo.index.json, and takes effect immediately for any running or future rabbit on that domain.
Important: This is a Rabbit Runner job, not a core engine job. The distinction matters — the core engine should never contain domain-specific filter logic. That knowledge lives in the index and is applied by the worker.
07
Director Command Extensions
These commands extend the existing ashDB command bus. They follow the same grammar as current commands (verb noun, space-separated).
spawn rabbit <repo-id>
Spawns a new rabbit worker for the named repo. Reads repo.index.json for strategy and config. Emits RABBIT_START. Shows progress in STREAM.
kill rabbit <id>
Stops a running rabbit. Saves checkpoint to sessionStorage for potential resume.
pause rabbit <id>
Pauses job queue processing. Rabbit holds its current state.
resume rabbit <id>
Resumes a paused rabbit from its last checkpoint.
status rabbits
Lists all active/paused/completed rabbits with job count, visited count, hits found.
add filter <domain> <pattern>
Adds a noise filter pattern to repo.index.json for the given domain. Takes effect immediately.
show index
Prints repo.index.json to STREAM — all known repos, strategies, noise filters.
show index <repo-id>
Prints the full entry for a single repo.
08
TOC Strategy — Bedrock Docs Walkthrough
Microsoft Learn publishes a toc.json for each documentation tree. Rabbit Runner flattens this tree into a URL list — this becomes the complete job queue before any fetching begins.
Step Description
Step 1 — Fetch TOC Rabbit fetches toc.json via the CORS proxy. Parses the nested tree, flattens all hrefs into absolute URLs. Applies noise filters. Result: clean URL list of every Bedrock doc page.
Step 2 — Seed priority Any URLs in the seeds array for this repo are moved to the front of the queue. Entity reference pages, block reference pages are fetched first.
Step 3 — Paginated fetch Rabbit works through the queue in batches of 5 (configurable). Each fetch goes through the proxy. Results scored against query, added to hits if relevant.
Step 4 — Checkpoint After every 10 fetches, rabbit writes its queue state and visited set to sessionStorage. If interrupted, resume picks up from here.
Step 5 — Complete When queue is empty, rabbit emits RABBIT_COMPLETE with stats: pages fetched, hits found, noise URLs discarded, duration.
09
Scope — What This Is Not Yet
This document is an architectural earmark. It describes what Rabbit Runner will be, not what exists today.
Current state:
  • traverseProxy() in the HTML is the interim implementation — single strategy, no index, no filtering
  • The proxy (proxy.js) is the transport layer that Rabbit Runner will use — it does not change
  • repo.index.json does not exist yet — it will be created alongside the first Rabbit implementation
  • Director command extensions for rabbit management are not yet wired

When Rabbit Runner is implemented, traverseProxy() will be replaced by a spawn call. The interface stays the same — DEEP_CRAWL button, STREAM output, hits in the panel. The worker layer is invisible to the user.
10
Relationship to Other Components
Component Relationship
proxy.js Transport. Rabbit Runner calls the proxy for every fetch. The proxy has no knowledge of repos, strategies, or filters — it just fetches and returns. This separation stays.
ashDB.js Observation and command layer. Receives RABBIT_* events and displays them in STREAM. Provides the command bus that Director uses to control rabbits. Does not execute jobs.
ashlight_v31 HTML Display and collection layer. Receives hits from Rabbit Runner via the same state.hits array. No changes to rendering logic needed.
repo.index.json Shared knowledge base. Written by Director, read by Rabbit Runner. The bridge between human intent and machine execution.
Bedrock vertical The first complete Rabbit configuration. bedrock_docs entry in the index, toc strategy, entity reference seeds, noise filters for Azure/Edge. Template for all future verticals.