Long-Context Degradation

Main finding. Qwen3.5-35B-A3B can retrieve a literal UUID at 1M tokens, yet its ability to bind and use distributed information collapses near the maximum YaRN extension. Nominal context capacity and effective context capacity are therefore materially different.

Purpose

This report evaluates how reliably Qwen3.5-35B-A3B uses information as the input grows from 8K tokens to the native 262K window and then to approximately 1M tokens with YaRN. The test suite deliberately progresses from literal lookup to tasks that require semantic matching, multi-step binding, revision selection, aggregation, and long grounded generation.

Evaluation ladder

Scope of the conclusions

• The 1M capability collapse is consistent across several independent task families.
• The repeated weakness near 70% of the input is treated as a working hypothesis, not a statistically established blind zone.
• Three seeds are sufficient for an engineering signal but not for a precise position-by-position capability estimate.

1. Setup and literal access

A controlled deployment baseline and a deliberately easy sanity check.

Experimental setup

Case 1 - Literal needle in a haystack

Prompt. Synthetic filler is generated to the requested context length and one labeled control record containing a random UUID is inserted near 10%, 50%, or 90% of the input.

Task. The model is asked to return only the UUID from that record; no paraphrasing, distractor resolution, or multi-step reasoning is required.

Score. A run passes only when the returned string exactly matches the inserted UUID.

... unrelated synthetic filler ...
CONTROL_RECORD: access_key = "7be4-91af-2c60"
... unrelated synthetic filler ...
Question: Return the exact access_key.

Literal retrieval accuracy by relative depth

Takeaway. Literal retrieval remains perfect at every tested length, including 1M. This sanity check confirms access, but it substantially overestimates useful long-context ability.

2. Semantic multi-needle retrieval

Five target facts, forty close distractors, and two absent facts expose a middle-context failure.

Prompt. Each seed contains five labeled source documents with target facts, placed near 10%, 30%, 50%, 70%, and 90% of the input, plus 40 semantically similar distractor documents. The questions paraphrase the targets instead of repeating their identifying wording; two additional questions refer to facts that are not present.

Task. The model must map each question to the supporting document ID, or return not_found.

Score. Positive queries require the exact document ID; absent queries are scored separately as abstentions.

DOC-D17: The Orion relay is protected by cobalt seals.
DOC-D28: The Orion relay reports through the western channel.
Question: Which document identifies the material used to protect Orion?
Expected: {"document": "DOC-D17"}

Exact source-document retrieval by position

Interpretation

• The 1M run loses every fact positioned near 50% and 70%, while the 90% fact remains intact.
• Perfect abstention suggests that instruction following survives better than positive source binding.
• The pattern is consistent across three seeds, but the sample remains too small to establish a fixed positional law.

Takeaway. Semantic access degrades sharply even though literal UUID retrieval remains perfect. The 50–70% failure is the first evidence for a possible middle-context blind region.

3. Multi-hop tracing and aggregation

The decisive capability split between non-thinking and thinking modes.

Prompt. Three independent chains are split across labeled documents: a mission names a relay, the relay names a vault, and the vault contains the terminal token. The same input also contains a 20-row ledger with mixed statuses and costs.

Task. For every mission, the model must follow both links and return the terminal token; it must also list every FAILED row, count them, and sum their costs.

Score. Hop documents, terminal tokens, complete chains, and the exact ledger set/count/sum are measured separately.

MISSION M17 -> relay R04
RELAY   R04 -> vault V92
VAULT   V92 -> token K7F2
LEDGER  J12 | status=FAILED | cost=430
Question: Resolve each token; list failed IDs, count them, and sum cost.

Non-thinking capability floor

Decision. Non-thinking mode failed the two-hop task already at the shortest evaluated length of 64K. This is a capability floor, not measurable context-length degradation. Subsequent complex experiments therefore use thinking mode only.

Thinking mode restores the task, then collapses near 1M (YaRN)

Measured at 8K, 32K, 64K, 229K and 977K. At 977K, no full chain or aggregation survives: only 2/9 terminal tokens and 8/27 individual hop documents are correct. The task is strong through 64K and remains largely solvable at 229K before failing at the maximum extension.

Takeaway. Thinking is essential for this task, but it does not prevent the extended-context collapse. The current data locate the transition somewhere between 229K and 977K.

4. Conflicts and updates

Identity resolution, revocations, competing revisions, exact values, and provenance.

Prompt. Five aliases point to canonical entities, while registry records for those entities are scattered elsewhere in the context. The records include competing revision numbers, non-authoritative sources, and explicit revocations; one additional alias has no valid match.

Task. Resolve alias to entity, discard invalid records, select the highest non-revoked authoritative revision, and return entity, revision, exact value, and source ID.

Score. All four fields must agree with the winning record; the absent alias must produce not_found.

ALIAS: "Northstar" -> entity E17
REGISTRY R31: E17 | revision=3 | value=blue | authoritative=true
REGISTRY R44: E17 | revision=4 | value=red  | status=REVOKED
Expected: {"entity":"E17","revision":3,"value":"blue","source":"R31"}

Exact alias-to-record resolution with YaRN

Measured at 8K, 32K, 64K, 229K and 977K. At the maximum extension:

Failure breakdown at 977K

Position-pair result

Takeaway. The model often emits a plausible revision number while losing the entity-to-record binding. Revision-number accuracy alone does not prove that the update rule was correctly executed. Alias and winner positions are also confounded by the fixed placement offset.

5. Long-context generation

Five distributed specifications must remain grounded and consistent across a long report.

Prompt. Five FINAL workstream specifications are placed at different depths, mixed with DRAFT records that reuse the same workstream IDs but contain conflicting owners, regions, and gates.

Task. Using FINAL records only, the model must write an executive table, five detailed sections of at least 180 words each, and the requested rollout order. Every core fact must appear consistently in both summary and detail.

Score. The scorer checks repeated fact coverage, section completeness, contradictions, invented IDs, and exact rollout order.

WORKSTREAM W3 | status=FINAL | owner=OPS-17 | region=NORTH | gate=G4
WORKSTREAM W3 | status=DRAFT | owner=OPS-08 | region=WEST  | gate=G2
Instruction: Use FINAL records only; repeat each fact in summary and detail.

Factual grounding and complete sections with YaRN

Measured at 8K, 32K, 64K, 229K and 977K. A complete workstream requires both exact facts and at least 180 words in its detailed section. Keeping the two metrics separate prevents short but factually correct responses from being misclassified as retrieval failures.

Position-level results at 977K

Native-window variability

Some 32K and 229K failures are global output-length or whole-response failures rather than position-specific retrieval failures. With only three stochastic samples, this creates non-monotonic results. At 977K, however, factual grounding itself disappears across most positions, producing a qualitatively different failure mode.

Takeaway. The model can generate a well-grounded long report at 229K, but near 1M it produces shorter, poorly grounded output. Only the workstream near 10% is fully recovered.

6. Cross-task synthesis

The useful context window is substantially smaller than the literal-access window.

Native-boundary performance versus maximum YaRN extension

Across all four complex task families - semantic retrieval, multi-hop, conflicts and generation - performance at the native boundary (229–262K) is far above performance at the maximum extension (977K–1M). Metrics are task-specific exact scores: positive document retrieval, full chains, exact alias resolution, and complete grounded workstreams. These are intended to show within-task degradation, not to rank task difficulty against each other.

Working positional hypothesis at the maximum extension

Working hypothesis. Information placed near 70% of a 1M-token input is consistently difficult to recover across the current task suite. The evidence is suggestive and reproducible within three seeds, but position, item identity, and task structure are not yet fully disentangled.

Conclusions

• Literal NIAH is an insufficient measure of effective long-context capability.
• Non-thinking mode reaches a capability floor on two-hop reasoning at 64K; harder tasks should be evaluated in thinking mode.
• Thinking mode remains strong through the native range but collapses across all complex task families near the 1M YaRN limit.
• Abstention and output-schema behavior survive better than positive retrieval and cross-record binding.
• The current report supports a strong 1M-collapse conclusion and a provisional 70%-position hypothesis.

References

1. Qwen Team. Qwen3.5-35B-A3B model card. huggingface.co/Qwen/Qwen3.5-35B-A3B
2. Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? 2024. arxiv.org/abs/2404.06654
3. Modarressi et al. NoLiMa: Long-Context Evaluation Beyond Literal Matching. 2025. arxiv.org/abs/2502.05167
4. LongGenBench: Long-context generation evaluation. 2024. arxiv.org/abs/2410.04199