• TensorTeach's Newsletter
  • Posts
  • RL Strengthens Circuits, DS-MoE Accelerates Reasoning, RL vs SFT Diverge, Rare Neurons Specialize, Styles Benchmark Thinking

RL Strengthens Circuits, DS-MoE Accelerates Reasoning, RL vs SFT Diverge, Rare Neurons Specialize, Styles Benchmark Thinking

Author

TensorTeach AI
September 26, 2025

Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs

What’s the research question?
How does reinforcement learning (RL) fine-tuning alter the internal circuitry of large language models (LLMs)?


What did the authors do?
The authors investigated the internal changes in LLMs caused by RL fine-tuning using a novel graph-theoretic approach:

  • Represented each LLM as a directed acyclic graph (DAG), where nodes are sub-modules (attention and feed-forward layers) and edges are residual connections.

  • Estimated the importance of each edge using Edge Attribution Patching (EAP), a gradient-based method that measures the impact of ablating edges on loss without multiple forward passes.

  • Analyzed four pairs of open-source LLMs (DeepSeek-Math, Mistral, Distilled-Qwen, and Qwen2.5), each with a base and RL-finetuned version.

  • Generated token sequences for a set of questions, filtered for correct answers, and truncated sequences to a fixed length for consistency.

  • Computed three key metrics across samples and layers:

    • Activation Intensity: average absolute edge weight, indicating overall pathway strength.

    • Information Complexity: Shannon entropy of edge weights, measuring diversity of internal pathways.

    • Distribution Kurtosis: shape of edge weight distributions, indicating how peaked or flat the importance distribution is.

What did they find?
RL fine-tuning consistently altered the internal circuitry of LLMs in meaningful ways:

  • Across all four model pairs, RL fine-tuning increased Activation Intensity, suggesting stronger internal pathways.

  • It also increased Information Complexity, indicating a richer diversity of active pathways and more nuanced internal representations.

  • At the same time, RL fine-tuning decreased Distribution Kurtosis, implying a shift from highly peaked importance distributions to more balanced ones.

  • Visualizations of edge importance changes showed many connections became more active after RL, highlighting a broad reorganization of internal circuitry.

  • Static preference-based RL approaches like DPO showed weaker or inconsistent effects, emphasizing the importance of the RL fine-tuning process itself.

Why does this matter?
This study provides a mechanistic understanding of how RL fine-tuning reshapes the internal workings of LLMs. By revealing that RL enhances both the strength and diversity of internal pathways, it offers insights into how RL improves model generalization and adaptability. These findings can guide the design of more robust, interpretable, and reliable LLMs, ultimately advancing their deployment in real-world applications such as AI assistants, educational tools, and complex reasoning tasks. Understanding internal circuitry changes also contributes to the broader goal of demystifying how large models learn and adapt.

Key Points

  • RL fine-tuning increases activation strength and diversity in LLM internal pathways.

  • Graph-theoretic analysis with Edge Attribution Patching quantifies internal changes without costly computations.

  • Enhanced internal circuitry correlates with improved model generalization and robustness.

  • Static preference-based RL methods show weaker effects compared to standard RL fine-tuning.

Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architectures

What’s the research question?
Can adaptive depth and expert specialization within transformer models enhance reasoning accuracy and computational efficiency?


What did the authors do?
- Developed DS-MoE, a transformer variant that dynamically allocates computational depth by composing specialized expert modules into reasoning chains.
- Implemented a learned routing network that evaluates input complexity using syntactic, semantic, and reasoning indicators to activate only the most relevant experts.
- Organized experts hierarchically into five categories: shallow (pattern recognition), medium (multi-step inference), deep (logical reasoning), memory integration, and meta-cognitive modules.
- Selected top-k experts for each input and composed them sequentially into reasoning chains, passing the input through these experts to generate output.
- Trained the model end-to-end with a joint loss function balancing task accuracy, routing fidelity, and load balancing.
- Used The Pile, an 800GB diverse corpus covering scientific papers, legal texts, programming code, and web content, for training and evaluation.


What did they find?
- Achieved up to 16% computational savings and 35% faster inference compared to uniform-depth transformers.
- Delivered 2.8% higher accuracy on multi-step reasoning benchmarks.
- Outperformed baseline models in accuracy, efficiency, and interpretability, especially on complex reasoning tasks such as legal and long-context datasets.
- Ablation studies confirmed the critical roles of routing, expert diversity, and specialized modules in performance gains.
- Limitations include potential complexity in expert design and routing optimization, which may require careful tuning for different tasks.


Why does this matter?
- Demonstrates that adaptive depth and expert specialization can significantly improve reasoning quality and computational efficiency in large-scale transformer models.
- Enhances model interpretability by explicitly modeling reasoning chains and input complexity.
- Offers a scalable approach suitable for resource-constrained environments, enabling deployment of powerful reasoning models without prohibitive costs.
- Aligns with cognitive science insights on human-like adaptive reasoning, paving the way for more intelligent and flexible AI systems.

Key Points

  • Introduces DS-MoE, a transformer variant with dynamic expert routing and adaptive depth.

  • Hierarchically organizes experts into pattern recognition, inference, reasoning, memory, and meta-cognitive modules.

  • Achieves up to 16% computational savings and 2.8% higher accuracy on reasoning benchmarks.

  • Improves interpretability by composing explicit reasoning chains based on input complexity.

RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs

What’s the research question?
How do reinforcement learning (RL) and supervised fine-tuning (SFT) differently influence the reasoning abilities of large language models (LLMs)?


What did the authors do?
The authors conducted a detailed, multi-level analysis of how RL and SFT shape reasoning in LLMs, focusing on two key aspects:

  • Trajectory-level analysis: They generated multiple reasoning outputs (trajectories) from models of different sizes (1.5B, 7B, 14B parameters) on math reasoning tasks. Using the chrF metric, they clustered similar outputs to identify unique reasoning paths, tracking how many correct and incorrect trajectories appeared before and after applying RL, SFT, or both.

  • Step-level analysis: They broke down reasoning outputs into sentences, embedded each sentence into a shared vector space, and clustered these embeddings into nodes to construct reasoning graphs. Edges connected consecutive reasoning steps. They analyzed these graphs using metrics like visitation frequency, degree, betweenness centrality, edge density, clustering coefficient, and modularity to understand how reasoning processes are organized and distributed.

What did they find?
The study revealed contrasting effects of RL and SFT on reasoning:

  • RL compresses incorrect reasoning paths: It reduces the number of unique wrong trajectories, effectively squeezing out erroneous reasoning strategies.

  • SFT expands correct reasoning paths: It increases the diversity and number of correct trajectories, encouraging exploration of valid reasoning strategies.

  • Step-level graph structure: RL increases the decay rate of node visitation frequency, degree, and betweenness centrality, consolidating reasoning into fewer, more central nodes. In contrast, SFT decreases these decay rates, dispersing reasoning across more nodes.

  • Global graph properties: RL creates hub-centric graphs with prominent central nodes, while SFT produces more globally connected graphs with distributed reasoning nodes. These patterns are consistent across model sizes and statistically significant.

Why does this matter?
This work advances our understanding of how different training strategies shape the reasoning capabilities of LLMs. By revealing that RL tends to focus reasoning into fewer, more central paths while SFT encourages diverse and expansive reasoning, it provides practical guidance for designing training regimes tailored to specific reasoning goals. The innovative analysis framework—combining trajectory clustering and reasoning graph analysis—offers a powerful tool for exploring interpretability, robustness, and the evolution of reasoning in AI systems. This insight is valuable for developing more effective AI assistants, improving AI reasoning robustness, and guiding future research on training large language models for complex cognitive tasks.

Key Points

  • RL compresses incorrect reasoning paths, reducing diversity but increasing focus.

  • SFT expands correct reasoning paths, promoting exploration and diversity.

  • Step-level analysis shows RL consolidates reasoning into fewer central nodes; SFT disperses reasoning across many nodes.

  • Global reasoning graph structures differ: RL creates hub-centric graphs; SFT yields more connected, distributed graphs.

Distributed Specialization: Rare-Token Neurons in Large Language Models

What’s the research question?
How do large language models internally implement mechanisms for processing rare tokens, which are words or symbols that appear infrequently in training data?


What did the authors do?
The authors conducted a comprehensive analysis of the internal neuron organization in several large language models, including GPT-2 and Pythia, across different sizes. Their approach included:

  • Identifying neurons with a disproportionate influence on rare token prediction by temporarily ablating (disabling) individual neurons and measuring the impact on model output using a metric called Δloss.

  • Classifying neurons based on their influence into a three-regime hierarchy: highly influential 'plateau neurons,' neurons with influence following a power-law decay, and minimally contributing neurons.

  • Analyzing activation patterns to assess whether influential neurons activate in a coordinated manner (low effective dimensionality) or independently.

  • Examining the spatial organization of neurons to see if influential neurons cluster together or are distributed.

  • Investigating whether rare tokens access specialized processing pathways via attention routing mechanisms.

  • Applying spectral analysis to weight correlation matrices to understand the underlying organization of influential neurons.

  • Repeating these analyses across multiple model architectures and scales to ensure robustness and generality.

What did they find?
The study uncovered several key insights into how large language models handle rare tokens:

  • Rare token processing is governed by a hierarchical influence structure: a small set of plateau neurons exert a dominant effect, followed by neurons with diminishing influence, and a large set of neurons with minimal impact.

  • Plateau neurons show coordinated activation patterns (low effective dimensionality), indicating distributed rather than modular specialization.

  • These influential neurons are spatially distributed rather than clustered, challenging the idea that specialized neurons need to be localized.

  • Rare tokens access specialized processing pathways through standard attention mechanisms without requiring dedicated routing or modular structures.

  • Spectral analysis revealed that influential neurons develop distinctive heavy-tailed weight correlation spectra, consistent with their functional specialization.

  • These organizational principles are consistent across different model sizes and architectures, suggesting a universal mechanism for rare token processing in LLMs.

Why does this matter?
This work provides the first systematic evidence that large language models implement distributed specialization for processing rare tokens, rather than relying on dedicated modules or routing pathways. This challenges common assumptions about how neural networks organize their internal functions and highlights the importance of distributed coordination within shared architectures. Understanding these principles can improve model interpretability, making it easier to analyze how models handle challenging inputs. It also offers new directions for enhancing model efficiency and robustness by leveraging the hierarchical influence structure and spectral signatures identified. Ultimately, these insights can inform the design of next-generation language models that are more controllable, scalable, and better at handling the full diversity of language—including the rare and complex tokens that are crucial for nuanced understanding and generation.

Key Points

  • Rare token processing in LLMs follows a three-regime influence hierarchy: plateau neurons, power-law neurons, and minimally contributing neurons.

  • Influential neurons are distributed and coordinated, not localized or modular.

  • Standard attention pathways, not dedicated routing, enable rare token specialization.

  • Spectral signatures reveal functional specialization among influential neurons.

StyleBench: Evaluating thinking styles in Large Language Models

What’s the research question?
How do different reasoning styles perform across various tasks and model architectures in large language models?


What did the authors do?
The authors developed StyleBench, a comprehensive benchmark to evaluate five distinct reasoning styles in large language models (LLMs):

  • Chain-of-Thought (CoT): Step-by-step reasoning

  • Tree-of-Thought (ToT): Search-based hierarchical reasoning

  • Algorithm-of-Thought (AoT): Algorithmic reasoning patterns

  • Sketch-of-Thought (SoT): Concise, structured reasoning

  • Chain-of-Draft (CoD): Drafting multiple reasoning paths

and five challenging tasks:

  • Mathematical reasoning

  • Question answering

  • Puzzle-solving

evaluating 15 open-source models ranging from 270M to 120B parameters. Each model was prompted with 500 examples per reasoning style and task, with responses automatically scored for correctness. Models were grouped into:

  • Small (<5B parameters)

  • Medium (5–15B)

  • Large (>15B)

What did they find?
Key findings include:

  • Performance scaled positively with model size, especially for search-based reasoning styles (ToT, AoT) on complex tasks like puzzle-solving and advanced math.

  • Concise reasoning styles (SoT, CoD) excelled on structured tasks that benefit from brevity and clarity.

  • Smaller models (<5B) often struggled to follow reasoning instructions, frequently defaulting to guessing rather than reasoning, highlighting limitations in their capacity to adopt complex thinking styles.

  • Larger models (>15B) demonstrated more robust and accurate reasoning across styles and tasks, showing the importance of scale for reasoning capabilities.

Limitations include the reliance on automatic evaluation metrics and the focus on open-source models, which may differ from proprietary LLMs like GPT-4.


Why does this matter?
This work provides a comprehensive framework for understanding how different reasoning styles interact with model size and task complexity in large language models. By revealing which styles work best for specific tasks and model scales, it guides the development of more adaptive and efficient reasoning systems. This has broad implications for improving AI performance in complex problem-solving, question answering, and reasoning applications, ultimately advancing the deployment of smarter, more reliable AI assistants.

Key Points

  • Introduces StyleBench, a benchmark for evaluating multiple reasoning styles in LLMs

  • Shows that reasoning style effectiveness depends on task complexity and model size

  • Large models outperform smaller ones, especially on search-based reasoning strategies

  • Provides insights to guide the design of adaptive reasoning approaches in AI