CosmicFish-HRM
Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models. A compact model that decides how hard to think before it answers.
Overview
Reasoning ability may depend not only on scale, but on how flexibly computation is used.
Large language models achieve strong reasoning, though often at the cost of massive parameter counts and expensive inference. A standard transformer applies the same depth of computation to every input, whether it is a short factual completion or a multi-step reasoning problem. CosmicFish-HRM takes a different view: some prompts are almost reflexive, while others require iteration and revision.
CosmicFish-HRM is a compact 82.77M parameter decoder-only model built around a Hierarchical Reasoning Module (HRM). Instead of fixed-depth computation, the model alternates between high-level and low-level reasoning states through iterative recurrent cycles. A learned halting head decides when enough reasoning has happened, so difficult inputs can trigger deeper reasoning trajectories while simpler prompts halt early and avoid wasted compute.
The architecture pairs this adaptive reasoning core with modern transformer components: Grouped Query Attention, Rotary Positional Embeddings, and SwiGLU activations. It was trained on CosmicSet, a curated 10B-token dataset covering web text, Wikipedia, code, mathematics, and research papers.
How it works
Three stages: learn to represent, reason adaptively, then predict.
Transformer foundation
Tokens map to dense embeddings (d = 448, vocabulary of 50,304) and pass through six input and six output pre-normalized transformer blocks, with the embedding matrix weight-tied to the language modeling head.
Grouped Query Attention shares key-value heads across query groups (8 query heads, 4 KV heads), cutting KV-cache memory while keeping most of multi-head attention's capacity. RoPE encodes relative position directly in attention, and SwiGLU feedforward layers use a gate to scale activations.
Dual recurrent states
The HRM core sits between the transformer stacks and maintains two interacting recurrent states, both initialized from the input stack's representation:
A high-level state H handles slower, more abstract planning. A low-level state L handles finer-grained local computation. At each reasoning step, the low-level module runs two cycles conditioned on the high-level state, then the high-level module runs two cycles conditioned on the updated low-level state, forming a feedback loop between abstraction and detail.
Learning when to stop
After every reasoning step, a lightweight halting head projects the mean-pooled high-level state to two scores: halt and continue. The model stops reasoning once the halt score exceeds the continue score, so no hand-crafted stopping rule is needed.
During training, an exploration mechanism (10% probability) defers halting to a random step between 2 and the 16-step maximum, exposing the network to longer reasoning trajectories. At inference, a small bias of 0.35 toward halting encourages efficiency while still allowing deep trajectories for hard inputs.
Key specs
A compact architecture built to think flexibly.
Results & tradeoff
At compact scale, adaptive reasoning is an architectural tradeoff, not a free lunch.
| Model | HellaSwag | PIQA | WinoGrande |
|---|---|---|---|
| CosmicFish-90M | 27.9 | 59.8 | 50.6 |
| CosmicFish-120M | 26.9 | 59.2 | 50.7 |
| GPT-2 Small (117M) | 29.7 | 62.5 | 50.7 |
| OPT-125M | 30.6 | 62.6 | 52.9 |
| Pythia-160M | 29.4 | 62.1 | 52.8 |
| CosmicFish-HRM | 26.2 | 58.1 | 50.7 |
| Benchmark | Mean steps | Std. dev. |
|---|---|---|
| HellaSwag | 3.033 | 6.263 |
| PIQA | 1.866 | 5.132 |
| WinoGrande | 0.954 | 3.777 |
| TriviaQA | 0.804 | 3.017 |
| Overall | 2.681 | 5.949 |
The tradeoff
At compact scale, CosmicFish-HRM does not outperform conventional transformers of similar size. A portion of its parameter budget and compute is dedicated to the recurrent reasoning infrastructure, halting behavior, and hierarchical state interaction. With fewer than 100M parameters, that overhead is a relatively large fraction of total capacity.
The paper's hypothesis is that this tradeoff changes as scale grows: the HRM core stays small relative to the backbone, so its proportional cost shrinks while the adaptive behavior remains. Compact-scale experiments are best read as testing whether adaptive reasoning can emerge at all inside autoregressive language models, rather than whether it immediately wins shallow benchmark accuracy.
Adaptive behavior in action
The model does not collapse into a single reasoning depth. Harder prompts think longer.
“The capital of France is”
“Photosynthesis is the process by which plants”
“If all roses are flowers and some flowers fade quickly, what can we conclude about roses?”
“A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?”
Short factual completions generally halt after a few iterations, while prompts involving abstraction, chained inference, or cognitive reflection trigger deeper trajectories. The bat-and-ball prompt, widely used in cognitive psychology to separate fast intuitive answers from slower deliberate reasoning, draws substantially more compute than simple factual recall. The halting head is responding to meaningful differences in prompt structure, not behaving randomly.
This behavior emerges without manually designed stopping rules. It arises from the interaction between the learned halting head, the recurrent reasoning states, and a lightweight step penalty (λ = 0.01) in the training objective.
Limitations & future work
An initial architectural exploration, not a final solution.
Deeper thinking is not correctness
More reasoning steps do not guarantee right answers. The paper documents failure cases, including factual hallucination and arithmetic errors such as “5 + 5 = 25”. And because most benchmarks measure short-context prediction and shallow commonsense, they are not designed to expose long-horizon iterative reasoning.
Scaling & reasoning evaluation
Open questions include how HRM-style reasoning behaves at larger scale, where the overhead shrinks relative to the backbone; evaluating on tasks requiring long chains, planning, and math; and combining adaptive depth with sparse Mixture-of-Experts routing for scalable capacity and compute.
Read it yourself
The paper, the weights, and the code are all open.