SULTAAN.AI

Architecture

Inside the kernel

One Python package, one native Rust module, several internal Rust crates, and a parallel track for training a model from nothing. Here is what each part does today.

Plan

The shape of it

How the Sultaan kernel is layered From top to bottom: the sultaan command line; the Python orchestration layer with the agent loop, memory, retrieval, knowledge graph, verifier and sandbox; the Backend protocol; two backends, llama.cpp and the sultaan native module built as one PyO3 wheel; and under the native module three Rust crates: sultaan-core, sultaan-inference and sultaan-tokenizer. A separate PyTorch pretraining track trains the model from scratch with atomic, resumable checkpoints. The Docker sandbox is built but not yet wired into the command line. You · the sultaan CLI PYTHON · ORCHESTRATION Agent loop and tools Memory · SOUL.md · knowledge graph Hybrid retrieval · grounding verifier Docker sandbox (built, not yet wired in) BACKEND PROTOCOL llama.cpp quantised GGUF models sultaan._native one PyO3 wheel RUST · HOT PATHS sultaan-core Backend trait, shared types sultaan-inference Candle backend sultaan-tokenizer HF tokenizers PARALLEL TRACK · PYTORCH From-scratch pretraining Decoder · streamed data · trainer Atomic checkpoints, resume after a crash
The layers, top to bottom. The native module re-exports the Rust crates, so there is one wheel to build.

The most important early commitment was a single Python wheel with a single native module. Several separate native wheels collide on Windows; one wheel avoids that, the same pattern used by Polars and Hugging Face tokenizers. Rust is reserved for hot paths, Python handles orchestration, and a small xtask runner keeps developer scripts identical on Windows and Linux.

Chambers

Each part, in plain words

Inference

A Backend trait in Rust is the contract every engine must meet, mirrored by a Python protocol so any adapter fits without inheritance.

Candle backend
Rust implementation of Qwen2-family generation, running on CPU today, reached from Python through the native module.
llama.cpp backend
The escape hatch for quantised GGUF formats that Candle does not support.
Bootstrap model
Hello-world runs on the open Qwen2.5-0.5B-Instruct model while Sultaan’s own model is trained.

The agent

The loop is a plain set of data classes: a system message, user turns, tool calls, tool results and a final answer, with a circuit breaker and structured logs. Its tools are a Python REPL, a retrieval tool and a knowledge-graph tool. The REPL runs on the host machine, so it is for trusted use only.

Memory and identity

SOUL.md
A creator-written identity file with standing instructions, parsed at session start and folded into the system message.
Conversational memory
A local-only wrapper around mem0, with a guard against sending memories to the cloud: recall before each turn, remember after it.
Knowledge graph
A SQLite subject-predicate-object store with trust tiers for facts.

Retrieval and grounding

Documents in a folder are chunked and indexed with BM25 keyword search plus optional dense vectors, and the two rankings are fused with reciprocal rank fusion. A grounding verifier then checks the answer sentence by sentence against what was retrieved, flags anything unsupported and attaches the best supporting source. An executable test holds it to catching at least 80% of planted hallucinations.

The sandbox

For code the project does not trust there is a Docker container with no network access, Linux capabilities dropped and limits on CPU, memory and time. It is built and unit-tested, but no command uses it yet, and its real escape test needs a running Docker daemon. Until it is wired in, the agent’s Python REPL runs on the host and is not meant for untrusted input.

Evaluation

An offline suite benchmarks any model, including every pretraining tier: teacher-forced perplexity, a subset of MMLU, a subset of TruthfulQA and a set of hallucination probes.

Pretraining

Model
A Llama-style decoder (RMSNorm, rotary position embeddings, SwiGLU, grouped-query attention) using PyTorch’s built-in scaled dot-product attention, so no extra attention library is required.
Data
FineWeb-Edu streamed from the Hugging Face Hub and packed into full windows, so there is no multi-terabyte download. Local corpora such as public-domain books are supported too.
Trainer
AdamW or 8-bit AdamW, cosine learning-rate schedule, bf16, gradient checkpointing, atomic checkpoints with resume, a NaN guard and wall-clock budgets.
Tokeniser
Starts from the Qwen2.5 tokeniser; a smaller custom BPE can be trained with the same toolkit, as the first-contact run did.

Not yet multi-GPU safe. The model is wrapped for FSDP, but every rank still sees the same batches and writes the same checkpoint folder. Training is single-GPU until that is fixed.

Workshop

How it is built

  • Toolchain: uv for Python, maturin to build the Rust extension, and a pinned Rust compiler, because the native ABI can break across compiler versions.
  • Versioning: DVC for models and datasets, so large files stay out of git.
  • Experiment tracking: mlflow.
  • Hygiene: pre-commit hooks for ruff, black, cargo fmt, clippy and secret detection; Conventional Commits; an ADR for any new dependency.

The command line

These commands are taken from the project README. The repository is private, so they are shown for illustration, not as an install guide.

sultaan generate --backend candle "Hello, who are you?"
sultaan pretrain run --smoke
sultaan rag query ./docs "hybrid search" --top-k 5
sultaan kg add facts.db Paris capital_of France
sultaan agent --rag ./docs --verify "What does ADR 0017 decide?"

What comes nextThe first-contact run