The AI Sleep Cycle: Architectural Path to High-Context Local Personalization

Every AI assistant today faces a fundamental bottleneck: Context Rot.

As conversations extend into days, weeks, and months, developer systems attempt to maintain continuity by stuffing thousands of lines of chat history into the prompt window. While modern LLMs brag about context windows spanning millions of tokens, the operational reality is grim: attention decays, retrieval accuracy drops, latency spikes, and the model's reasoning capabilities degrade.

At Last Machine, we are exploring alternative pathways to continuous learning. Instead of dragging a massive bag of historical text through every prompt, we built Ameles—an architecture that lets a small, local model "sleep" and "dream" overnight, distilling its daytime experiences directly into its neural weights.

Here is the technical architecture under the hood.

1. Bypassing Context Rot with Weight-Based Memory

When an LLM depends entirely on its in-context history, its performance is bounded by the attention mechanism's limits. It must recalculate attention maps across a expanding history on every single token generation.

Our approach separates immediate working memory from long-term personalization:
* Daytime (Short-Term Memory): The assistant runs a fast, lightweight local model (microsoft/Phi-4-mini-instruct, quantized to 4-bit) for terse, direct interactions.
* Nighttime (Consolidation): Rather than preserving this history in-context forever, the system spins up a nightly pipeline that processes daytime logs, extracts lessons, and fine-tunes the local weights.

By transferring episodic chat logs into structural parametric memory, we keep the active context window clean. The result is a model that remains small, lightning-fast, and deeply personalized.

2. The Smart API Proxy Gateway

To capture training data from standard chat interfaces without manual intervention, Ameles implements an OpenAI-compatible API proxy gateway written in FastAPI (chat.py).

[ Hermes Agent / Client ] 
          │
          ▼
    [ API Proxy ] ───( Logs 'ameles' traffic to JSONL )
          │
          ▼
   [ Ollama Engine ]

When external desktop clients (like Hermes Agent) communicate with the local model, they send queries to the Ameles port (8001). The proxy acts as an intelligent middleman:
* Targeted Filtering: To prevent training-set contamination, the proxy only logs queries targeting the model identifier "ameles". If you are running massive development prompts through other Ollama models (e.g., hermes3:8b), the proxy passes the traffic straight through to Ollama unlogged.
* Structured JSONL Logs: Conversations with "ameles" are parsed and written as flat, single-line JSON objects to sleep_cycle/day_logs/<date>.jsonl. This schema records the raw prompts, model responses, and any manual user corrections or escalations.

3. The Multi-Phase Sleep Pipeline

The training lifecycle is divided into a series of highly isolated, stateful phases, which can be monitored in real-time from the dashboard:

idle ──► collecting ──► reviewing ──► dreaming ──► training ──► converting ──► promoting ──► done

Phase A: Collecting and Filtering

The pipeline gathers raw logs from the last 7 days of chat history. It splits the exchanges into a training set and an independent validation set (typically holding out 20% of the exchanges to evaluate training progress).

Phase B: Teacher Review and Synthetic Expansion (Dreaming)

To prevent garbage-in, garbage-out failures during fine-tuning, Ameles uses a larger "Mentor" model (such as anthropic/claude-sonnet-4.5 via OpenRouter):
1. Review: The mentor performs a keep / fix / expand / drop pass on each conversation log. If the local model hallucinated or gave a subpar answer, the mentor fixes it, providing a high-quality target label.
2. Dreaming: For every verified example, the mentor generates multiple paraphrased variations. This synthetic expansion simulates "lucid dreaming"—allowing the small model to practice recalling the same core facts from multiple linguistic angles.
3. API Cost Isolation: While active chat escalations can use live web search via OpenRouter's :online suffix (costing only ~$0.007 per query), web search is strictly disabled during the dreaming phase. Because dreaming evaluates static, historical facts, this design choice keeps overnight API costs negligible.

Phase C: Local Fine-Tuning (QLoRA Mechanics)

Once the dataset is prepared, the actual training loop begins. The system loads the target Phi-4-mini model and executes parameter-efficient fine-tuning (PEFT) using QLoRA (Quantized Low-Rank Adaptation).

Crucially, this phase runs 100% locally on a consumer-grade NVIDIA GeForce RTX 4070 SUPER GPU. It proves that personalized fine-tuning can be run effectively on consumer hardware, free of massive data centers or cloud subscription fees.

Under the hood, the training pipeline in go_to_sleep.py configures PEFT with strict mathematical parameters to balance personalization against resource constraints:

  1. 4-Bit Quantization (QLoRA): The base model (microsoft/Phi-4-mini-instruct) is loaded in 4-bit precision via bitsandbytes (bnb_4bit_compute_dtype=torch.bfloat16). This shrinks the model's footprint in VRAM to around ~2.5 GB, leaving ample headroom on a 12 GB RTX 4070 SUPER for batch calculations, gradients, and optimizer states.
  2. LoRA Hyperparameters ($r=16, \alpha=32$): Instead of modifying the base model's billions of parameters, LoRA freezes the original weights and injects pairs of low-rank decomposition matrices. We configure the Rank ($r$) to 16 and the Scaling Alpha ($\alpha$) to 32 with a dropout rate of 0.05. This restricts training to just a tiny fraction of total weights, reducing memory overhead and preventing catastrophic forgetting.
  3. Targeting Attention Projections: The training targets the core linear projections inside the self-attention mechanism:
    python target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]
    By focusing updates on the query, key, value, and output attention projections, the model learns how to pay attention to your specific conversational nuances and response formatting without losing its general reasoning capabilities.
  4. Continuous Learning (Incremental Adapters): If an adapter from a prior sleep cycle exists, the script automatically resumes training from those existing weights:
    python model = PeftModel.from_pretrained(model, prev_adapter, is_trainable=True)
    This allows for continuous incremental adaptation—the AI builds a rolling history of learning rather than throwing away yesterday's weights.

Phase D: Conversion, Weight Merging, and Hot-Promotion

Once training completes, the system enters the compilation phase:
1. Weight Merging: The script loads the base model in full precision, attaches the newly trained LoRA adapter, and merges them using PEFT (model.merge_and_unload()).
2. GGUF Compilation: The merged full-precision weights are exported, converted into GGUF format via llama.cpp scripts, and quantized back to 4-bit to optimize daytime execution speeds.
3. Hot-Promotion: The system runs ollama create ameles -f ameles-ctx.modelfile to register the new model. The proxy server then dynamically promotes the new model weights. The assistant is updated, awake, and ready for use by morning.

4. Bridging to Hermes Agent

In production, we integrate Ameles directly into Hermes Agent. By setting up a Custom Endpoint in Hermes to talk to http://127.0.0.1:8001/v1, we gain access to the proxy logging system natively.

Furthermore, we deployed a custom skill, ameles-control, to the Hermes directory. This bridges the runtime interface to the background server, allowing us to manage the training cycle directly from our conversation:
* Inspecting live server statuses (GET /api/status).
* Reviewing the mentor's dataset evaluations (GET /api/sleep/review).
* Triggering the sleep cycle (POST /api/sleep/start).

5. Why Researching Local AI Lifecycles Matters

The future of personal computing isn't centralized clouds containing trillion-parameter giants. The future is small, hyper-specialized, local models that live alongside you, securely recording and learning from your workflows.

By building systems that can sleep, dream, and restructure their own weights on consumer-grade silicon, we are taking a major step toward practical, sustainable, and highly private Super Intelligence.

Research is hard, but building the infrastructure that makes it possible is why we are here.