Chambers
Each part, in plain words
Inference
A Backend trait in Rust is the contract every engine must meet, mirrored by a Python protocol so any adapter fits without inheritance.
- Candle backend
- Rust implementation of Qwen2-family generation, running on CPU today, reached from Python through the native module.
- llama.cpp backend
- The escape hatch for quantised GGUF formats that Candle does not support.
- Bootstrap model
- Hello-world runs on the open Qwen2.5-0.5B-Instruct model while Sultaan’s own model is trained.
The agent
The loop is a plain set of data classes: a system message, user turns, tool calls, tool results and a final answer, with a circuit breaker and structured logs. Its tools are a Python REPL, a retrieval tool and a knowledge-graph tool. The REPL runs on the host machine, so it is for trusted use only.
Memory and identity
- SOUL.md
- A creator-written identity file with standing instructions, parsed at session start and folded into the system message.
- Conversational memory
- A local-only wrapper around mem0, with a guard against sending memories to the cloud: recall before each turn, remember after it.
- Knowledge graph
- A SQLite subject-predicate-object store with trust tiers for facts.
Retrieval and grounding
Documents in a folder are chunked and indexed with BM25 keyword search plus optional dense vectors, and the two rankings are fused with reciprocal rank fusion. A grounding verifier then checks the answer sentence by sentence against what was retrieved, flags anything unsupported and attaches the best supporting source. An executable test holds it to catching at least 80% of planted hallucinations.
The sandbox
For code the project does not trust there is a Docker container with no network access, Linux capabilities dropped and limits on CPU, memory and time. It is built and unit-tested, but no command uses it yet, and its real escape test needs a running Docker daemon. Until it is wired in, the agent’s Python REPL runs on the host and is not meant for untrusted input.
Evaluation
An offline suite benchmarks any model, including every pretraining tier: teacher-forced perplexity, a subset of MMLU, a subset of TruthfulQA and a set of hallucination probes.
Pretraining
- Model
- A Llama-style decoder (RMSNorm, rotary position embeddings, SwiGLU, grouped-query attention) using PyTorch’s built-in scaled dot-product attention, so no extra attention library is required.
- Data
- FineWeb-Edu streamed from the Hugging Face Hub and packed into full windows, so there is no multi-terabyte download. Local corpora such as public-domain books are supported too.
- Trainer
- AdamW or 8-bit AdamW, cosine learning-rate schedule, bf16, gradient checkpointing, atomic checkpoints with resume, a NaN guard and wall-clock budgets.
- Tokeniser
- Starts from the Qwen2.5 tokeniser; a smaller custom BPE can be trained with the same toolkit, as the first-contact run did.
Not yet multi-GPU safe. The model is wrapped for FSDP, but every rank still sees the same batches and writes the same checkpoint folder. Training is single-GPU until that is fixed.