CODA: Concept-Operation Decoupled Architecture for Knowledge-Transparent Reasoning

Summary. We validate the decoupling of reasoning from knowledge across four reasoning tasks: chain prediction, transitivity, sorting, and composition. A head-to-head comparison with a standard transformer of similar parameter count shows CODA is 34% faster at forward inference and uses 12% less peak memory, while retaining the unique advantage of zero-shot domain transfer.

1. Introduction

Large language models have demonstrated remarkable reasoning capabilities, but their knowledge is statically embedded in their parameters during training. This coupling of reasoning and knowledge creates fundamental limitations: updating outdated information requires expensive retraining, domain adaptation is inefficient, and there is no principled way to control which facts the model relies on during inference.

Prior work has attempted to address this through retrieval-augmented generation (RAG) [6] and differentiable memory architectures [2, 8, 10]. RAG retrieves raw text chunks from a corpus, introducing latency and opaque reliance on retrieval quality. Differentiable memory networks learn to read from and write to an external memory, but the memory mechanism itself is trained end-to-end, entangling the storage and retrieval process with the model's parameters.

CODA takes a different approach. We introduce a clean separation between an operational core that learns only reasoning operations and a knowledge memory that stores structured facts as key-value pairs. The operational core accesses memory through a cross-attention mechanism, but crucially, the memory module is constructed independently of training via an encoding process. This means:

• Knowledge independence. The model's reasoning accuracy depends almost entirely on the presence of memory. Removing the module collapses accuracy from 63.7% to 1.4%.
• Swappable knowledge. Different knowledge bases can be loaded at inference time with identical performance and no retraining.
• Transparent fact usage. The facts available to the model are explicitly enumerated in the memory module, enabling verification and audit.

2. Related work

2.1 Memory-augmented neural networks

Memory Networks [10] and End-to-End Memory Networks [8] introduced external memory for question answering. These architectures learn to attend over stored facts but require end-to-end training of the memory representation. Similarly, Differentiable Neural Computers [2, 3] learn read/write operations on external memory, but the memory content and access patterns are jointly optimised. In contrast, CODA's memory is constructed independently via a fixed encoding procedure and never updated during training - the model learns only how to query it.

2.2 Retrieval-augmented generation

RAG [6] and related approaches augment language models with dense passage retrieval from a corpus. While effective for knowledge-intensive tasks, RAG systems retrieve raw text and rely on the model to extract and reason over the retrieved information. This creates several limitations: the retrieval process is a separate, trained component; there is no structured representation of facts; and the model may ignore or override retrieved information with its parametric knowledge. CODA replaces raw-text retrieval with structured key-value lookups and uses curriculum learning to wean the model off parametric knowledge entirely.

2.3 Cross-attention as a universal interface

Recent theoretical work has shown that the feed-forward network in a transformer can be understood as a form of cross-attention over learned key-value pairs: FFN(x) = V · softmax(K · x) [4]. This suggests that cross-attention can serve as a general-purpose interface for conditioning neural networks on external information. Our work builds on this insight by replacing the FFN entirely with explicit cross-attention over an external knowledge memory, creating a clean architectural boundary between learned operations (self-attention + experts) and provided knowledge (memory).

2.4 Modular and decomposable architectures

Several lines of work have explored modular architectures where different components handle different aspects of processing [1, 5]. CODA extends this philosophy to the separation of knowledge from operations, demonstrating that a transformer can learn to reason abstractly when knowledge is provided as a structured external input. The mixture-of-experts routing within each layer [7] further modularises the reasoning operations themselves.

3. Architecture

CODA consists of three components operating at different stages of the pipeline. The operational core is a transformer whose per-layer FFN is replaced by cross-attention over an external knowledge memory; the knowledge memory is a structured key-value store built from symbolic triples via a fixed encoding procedure; the curriculum learning schedule gradually transfers facts from the prompt to the memory during training.

3.1 Operational core

The operational core is a transformer-based neural network that processes input sequences and produces output predictions. It is trained exclusively on reasoning operations, learning patterns like "if A precedes B in a chain, and the query asks what follows A, the answer is B." Critically, the operational core is trained without any embedded factual knowledge; all task-relevant facts are provided externally.

Each CODA layer follows the structure:

(1)  x ← x + SelfAttn(LN(x))
(2)  x ← x + CrossAttn(LN(x), memory)
(3)  x ← x + ExpertMix(LN(x))

Equation (1) handles relational reasoning across tokens. Equation (2) replaces the standard FFN with cross-attention to external memory, following [4]. Equation (3) applies specialised reasoning operations - relation inference, composition, transformation, and decision - with token-level routing selecting the top-2 experts per token.

Each cross-attention layer has an associated auxiliary prediction head that independently predicts the next token from the cross-attention output. These auxiliary heads (coefficient 0.5) provide direct gradient signal to every cross-attention layer, preventing dilution through the subsequent expert mixture and residual connections.

3.2 Cross-attention initialisation

The cross-attention projections - query, key, value, output - are initialised to identity matrices. This is a critical design choice: initially, the cross-attention behaves as a pass-through, allowing the model to first learn basic reasoning patterns from facts provided directly in the input context. The cross-attention weights then gradually depart from identity during Stage 2 curriculum training as the model learns to retrieve facts from memory.

3.3 Knowledge memory

The knowledge memory is a structured store of facts represented as key-value pairs. Each fact (s, r, o) - subject, relation, object - is encoded into a key vector (subject + relation) and a value vector (object). The encoding uses one of two procedures:

• One-hot encoding (fallback). Subject and relation are encoded as one-hot positions in a sparse vector of dimension d_mem, with entities occupying the first N positions and relations the next M.
• Embedding-based encoding (primary). The model's own token embedding weights are used to construct keys and values. For each entity or relation, the characters are looked up in the token embedding table and averaged: key_i = avg(s) + avg(r), value_i = avg(o).

The embedding-based encoding is essential because the cross-attention projections are initialised in the model's d_model space. When memory keys and values live in the same embedding space as the model's hidden states, the identity-initialised cross-attention can immediately perform meaningful retrievals.

This is a critical distinction from prior work: the memory is not trained. It is constructed from a set of domain facts using a transparent encoding, loaded at inference time, and can be swapped instantly between different domains.

3.4 Curriculum learning

Training proceeds in two stages:

1. Stage 1 (pretraining). The model learns fundamental reasoning patterns from examples where facts are provided directly in the input context. Cross-attention is initialised to operate as an identity function.
2. Stage 2 (curriculum). A curriculum gradually shifts facts from the input context into the external memory. A parameter ρ controls the probability that facts appear in the prompt, annealing from ρ = 0.9 (90% of examples have facts in the prompt) to ρ = 0.0 (no facts in the prompt) over approximately 27,000 training steps. The cross-attention layers are trained with a higher learning rate (1 × 10⁻⁴ vs. 3 × 10⁻⁵ for the core) to accelerate their adaptation.

This curriculum design is essential: without it, the model would continue to rely on in-context facts and never learn to use the memory interface effectively.

3.5 Entity prediction over binary classification

An early finding of this work is that binary classification tasks - "does A come before B? Yes/No" - cannot validate knowledge independence. In binary format, a model can achieve 50% accuracy by random guessing, and performance near 50% is consistent with either learned knowledge or random behaviour. We found that binary tasks showed no meaningful accuracy difference between memory-loaded and memory-free conditions for this reason.

All tasks are therefore framed as entity prediction: given a query with a wildcard (_), predict the missing entity from 26 possible answers (a single uppercase letter A–Z). This gives a 3.8% random baseline and provides clear signal for whether the model is genuinely using memory to retrieve the correct answer.

4. Experimental setup

4.1 Reasoning tasks

We evaluate on four synthetic reasoning tasks designed to require distinct reasoning operations:

• Chain prediction. Given a chain of relationships (A > B, B > C, C > D), predict the next entity following a given entity (A > _ → B). Direct lookup in sequentially ordered memory.
• Transitivity (reverse lookup). Given the same chain, find the entity that precedes a given entity (_ < C → B). Tests backward reasoning over chain structure, requiring retrieval and inversion of the relationship B > C.
• Sorting (max value). Given entities with associated numeric values (A=10, B=30, C=20), find the entity with the highest value. Tests comparison and aggregation over multiple memory entries.
• Composition (max result). Given entities with operations and values (A+5, B×2) and a target value, find which entity yields the highest result when its operation is applied to the target. Tests multi-step reasoning: retrieve each entity's operation, apply it, compare results, select the maximum.

All tasks use the prompt format Q: subj rel wildcard | A: - for example Q: A>_ | A: - with a single entity output, giving a random baseline of approximately 3.8% (1/26).

4.2 Model configuration

4.3 Training protocol

Stage 1 (30,000 steps). The model is trained from scratch with facts always in the prompt. Cross-attention initialised to identity. Optimiser: AdamW with linear warmup (500 steps) and cosine decay. Learning rate: 3 × 10⁻⁵ for core parameters, 1 × 10⁻⁴ for cross-attention parameters. Batch size 64.

Stage 2 (30,000 steps). The curriculum scheduler anneals ρ from 0.9 to 0.0 over 27,000 steps, then holds at 0.0 for the remaining 3,000. Embedding-based memory encoding is enabled using the model's token embedding weights. Other hyperparameters are identical to Stage 1. A per-layer auxiliary head (coefficient 0.5) provides direct gradient signal at each cross-attention layer.

4.4 Evaluation metrics

We report overall accuracy with memory loaded (a measure of reasoning capability); per-task accuracy; knowledge independence, being accuracy without memory, which should be near random; and swap consistency, being accuracy across different knowledge bases, which should be identical. All evaluations use 1,000 held-out examples per task (4,000 total) at ρ = 0.0 with a separate random seed from training.

4.5 Standard transformer baseline

For output quality comparison, we train a standard decoder-only transformer (d_model=512, 6 layers, 8 heads, 25M parameters) on identical data with facts always in the prompt. This represents an upper bound on single-domain performance, since facts are directly accessible in context rather than requiring retrieval.

For inference speed and memory benchmarking, we use a larger standard transformer (d_model=768, 12 layers, 12 heads, SwiGLU, 114M parameters) to roughly match CODA's parameter count.

5. Results

5.1 Reasoning capability

With memory loaded, the model achieves 63.7% overall accuracy against a 3.8% random baseline across all four tasks. All tasks are learned well above chance, confirming that the operational core can acquire and apply distinct reasoning operations through the memory interface. Chain prediction - the simplest task, requiring only direct entity lookup - achieves near-perfect accuracy, while multi-hop tasks show room for improvement.

5.2 Knowledge independence

When the memory module is removed - empty memory, no facts provided - accuracy collapses. Chain prediction drops from 99.2% to 0.4%, a 248-fold reduction. Sorting and composition fall to exactly 0%. The facts-in-prompt condition, where facts are provided in the input text but no memory module is present, achieves only 1.5%, confirming that the model cannot exploit in-context facts without the memory interface. It has genuinely learned to route knowledge retrieval through cross-attention rather than through direct sequence processing.

5.3 Knowledge swap

To test whether knowledge is truly swappable without retraining, we constructed two entirely different knowledge bases (KB A and KB B) and evaluated the same model on both, switching between them at inference time. Performance is identical across all configurations. There is zero degradation from swapping, no evidence of memory interference or cross-contamination, and the model returns to baseline immediately when the original KB is restored.

Experiment A - chain-only evaluation (single-step lookup).

Experiment B - all four tasks with multi-step chains.

5.4 Inference efficiency

We benchmarked CODA (90M parameters) against a standard transformer (114M) on the same GPU hardware. Despite having fewer total parameters, CODA is faster and more memory-efficient at all batch sizes: 34% faster at batch size 64, using 12% less peak memory. The speed advantage comes from CODA's smaller d_model (512 vs. 768) and the replacement of dense FFN layers with cross-attention plus sparse MoE experts.

5.5 Comparison with a standard transformer

We trained a standard decoder-only transformer (25M parameters, d_model=512, 6 layers, 8 heads, proper causal masking) on identical data with facts always in the prompt for 15,000 steps. This represents the conventional approach: a transformer that processes both facts and questions in its input sequence.

At comparable overall accuracy (~63.5%), the per-task profiles diverge significantly:

Key findings:

• Standard with facts-in-prompt outperforms CODA by 25.5pp overall. This is expected, since facts are directly accessible in the input rather than requiring retrieval.
• CODA's chain accuracy exceeds the standard even with facts in prompt (99.2% vs. 91.9%). The memory interface excels at direct entity lookup, suggesting that structured key-value memory is a more natural representation for lookup operations than sequential text.
• CODA's multi-hop reasoning lags significantly (transitivity 43.4% vs. 94.3%, sorting 50.6% vs. 89.3%). Retrieving and composing multiple facts from memory is intrinsically harder than reading them from contiguous text.
• At matched overall accuracy, per-task profiles are fundamentally different. CODA prioritises chain (99% vs. 36%) while the standard excels at multi-hop (52–84% vs. 43–61%). The architectures learn different inductive biases.
• Knowledge independence remains CODA's unique advantage. The standard transformer cannot swap domains without retraining. CODA swaps knowledge bases in a single inference call with zero degradation.

6. Discussion

6.1 Why knowledge independence matters

The central claim of this work is that reasoning and knowledge can be cleanly decoupled in neural networks. Our results support this with three converging pieces of evidence:

1. Accuracy depends on memory: 63.7% with memory, 1.4% without it.
2. Swapping causes no degradation: the same weights produce identical accuracy on different knowledge bases.
3. Baked-in knowledge is negligible: the facts-in-prompt baseline (1.5%) is indistinguishable from the no-memory baseline (1.4%).

The practical implications include zero-shot domain adaptation, instantly updatable knowledge, and transparent, auditable fact usage.

6.2 The chain advantage

CODA's exceptional chain prediction accuracy (99.2% vs. 91.9%) warrants analysis. Chain prediction is essentially a lookup operation: given subject S and relation R, retrieve the correct object O. This maps directly onto cross-attention: the model's query (S + R) attends over memory keys and retrieves the value (O). The structured key-value format is a more natural representation for this operation than the sequential text processing required by a standard transformer.

This suggests that CODA's cross-attention interface is not merely a substitute for in-context processing but offers a genuine architectural advantage for lookup-intensive reasoning. The performance gap on multi-hop tasks likely reflects the difficulty of composing multiple retrieval operations, which may benefit from iterative memory access or multi-hop attention mechanisms.

6.3 Limitations and future work

Task complexity. The current experiments use synthetic tasks with simple symbolic entities and relations. Scaling to natural language and real-world knowledge bases requires parsing unstructured text into structured triples.

Multi-hop reasoning gap. Closing the gap on transitivity, sorting, and composition is the primary direction for future work. Promising approaches include iterative memory access, improved memory encoding that preserves relational structure, and extended curriculum training.

Memory encoding. The current embedding-based encoding - averaging character embeddings - is a simple heuristic. Learned encoders or pretrained entity embeddings could improve memory representations. The identity-initialised cross-attention also requires d_model = d_mem, which may limit scaling flexibility.

Model scale. The operational core is relatively small (90M) compared to modern LLMs. Initial scaling experiments at 718M parameters showed that naive scaling without corresponding increases in data diversity and training steps leads to underfitting, suggesting that careful scaling recipes are needed.

Dynamic memory. Extending to dynamic memory updates during inference, multi-source knowledge integration, or confidence-weighted fact retrieval are natural future directions.

7. Conclusion

CODA demonstrates that neural networks can learn reasoning operations independent of the factual knowledge they apply them to. Across four distinct reasoning tasks, the model achieves strong accuracy with memory (63.7% overall) and collapses to near-random without it (1.4%). Knowledge bases can be swapped at inference time with no retraining and no performance degradation - 87.5% on both KB A and KB B.

Compared to a standard transformer of similar size, CODA is 34% faster at inference, uses 12% less peak memory, and, crucially, maintains the ability to change domains instantly without retraining, a capability that no standard transformer can match. This decoupling addresses fundamental limitations in current approaches to knowledge integration in neural networks, offering a path toward more transparent, updatable, and domain-adaptable reasoning systems.

References

1. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. CVPR, 2016.
2. A. Graves, G. Wayne, and I. Danihelka. Neural Turing machines. arXiv:1410.5401, 2014.
3. A. Graves et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016.
4. D. Guo and Y. Chen. FFNs are attention: Uncovering the implicit memory of transformers. arXiv:2501.00823, 2025.
5. L. Kirsch, J. Kunze, and D. Barber. Modular networks: Learning to decompose neural computation. NeurIPS, 2018.
6. P. Lewis et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS, 2020.
7. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR, 2017.
8. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-to-end memory networks. NeurIPS, 2015.
9. A. Vaswani et al. Attention is all you need. NeurIPS, 2017.
10. J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv:1410.3916, 2014.