# Smart Token Compression

LLM tokens are expensive, and verbose tool output is where most of them go to die. A `git status` in a busy repo, a `cargo build` log, a 600-message email thread, a `docker ps -a` against a real cluster, each of these can balloon a context window for almost no information gain.

OpenHuman ships with **TokenJuice**, a port of [vincentkoc/tokenjuice](https://github.com/vincentkoc/tokenjuice) integrated directly into the tool-execution path. Before any tool result reaches the model, TokenJuice runs the output through a rule overlay that strips the noise and keeps the signal.

## Three-layer rule overlay

Rules are JSON, and they merge in this order, later layers override earlier ones:

<table><thead><tr><th width="134.41796875">Layer</th><th>Path</th><th>Purpose</th></tr></thead><tbody><tr><td><strong>Builtin</strong></td><td>shipped with the binary</td><td>sensible defaults for git, npm, cargo, docker, kubectl, ls, etc.</td></tr><tr><td><strong>User</strong></td><td><code>~/.config/tokenjuice/rules/</code></td><td>your personal overrides, apply across every project</td></tr><tr><td><strong>Project</strong></td><td><code>.tokenjuice/rules/</code></td><td>repo-specific overrides, checked in, shared with the team</td></tr></tbody></table>

Each rule names a tool/command pattern and a reduction strategy (truncate, dedup lines, fold whitespace, drop matching regexes, summarize sections, …). New rules are just JSON files; no recompile required.

## Why this matters for memory

TokenJuice is what makes [auto-fetch](/openhuman/features/obsidian-wiki/auto-fetch.md) economically viable. When the Gmail provider syncs a page of 200 messages, TokenJuice compacts each canonicalized email *before* it enters the model that builds summaries. The same applies to GitHub diffs, Slack channel dumps, and any other firehose source.

Concretely: ingesting your last six months of email through a frontier model costs single-digit dollars instead of hundreds.

## Where it lives in the pipeline

```
tool call result
      │
      ▼
TokenJuice (classify → match rule → reduce)
      │
      ▼
LLM context
```

Implementation: `src/openhuman/tokenjuice/` (`classify.rs`, `reduce.rs`, `rules/compiler.rs`, `tool_integration.rs`).

## Inspecting and overriding

* Drop a JSON file in `~/.config/tokenjuice/rules/` to add or override a rule globally.
* Drop one in `.tokenjuice/rules/` inside a repo to do the same per-project.
* Start the core with `RUST_LOG=openhuman_core::openhuman::tokenjuice=debug` to see what's matching and how much output is being trimmed.

## See also

* [Native Tools](/openhuman/features/native-tools.md). most heavy tool output flows through TokenJuice.
* [Memory Tree](/openhuman/features/obsidian-wiki/memory-tree.md). the downstream consumer of compressed output.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://tinyhumans.gitbook.io/openhuman/features/token-compression.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.