- TensorTeach's Newsletter
- Posts
- ReVisual-R1 Leads in Multimodal Reasoning, MMRB Introduces New Benchmark, Turbo Improves Table Analysis, RAP Enhances Data Efficiency, and SPARKLE Breaks Down Math Reasoning
ReVisual-R1 Leads in Multimodal Reasoning, MMRB Introduces New Benchmark, Turbo Improves Table Analysis, RAP Enhances Data Efficiency, and SPARKLE Breaks Down Math Reasoning
TensorTeach AI
June 06, 2025
Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
What’s the research question?
How can staged training strategies and novel reinforcement learning techniques improve the reasoning abilities of multimodal large language models (MLLMs)?
What did the authors do?
The authors developed and evaluated a new training approach for a 7-billion-parameter open-source MLLM called ReVisual-R1, focusing on enhancing multimodal reasoning capabilities through a structured, three-stage curriculum:
Stage 1: Cold Start with Text-Only Data — The model was initially trained on high-difficulty, text-only datasets containing explicit reasoning paths, complex textual examples, and multimodal questions with ground truth annotations to establish strong foundational reasoning skills.
Stage 2: Multimodal Reinforcement Learning (MRL) — Using the Group Relative Policy Optimization (GRPO) algorithm, the model was fine-tuned on multimodal data organized into groups. The training was enhanced with Prioritized Advantage Distillation (PAD), which filters uninformative samples and prioritizes updates based on advantage signals to improve stability and efficiency.
Stage 3: Text-Only Reinforcement Learning (TRL) — The model was further refined on high-quality, instruction-focused text-only data to improve linguistic fluency and reasoning, complementing the multimodal training.
The architecture was based on the Qwen-2.5-VL-7B-Instruct model, with detailed hyperparameters provided in the appendix.
What did they find?
ReVisual-R1 achieved state-of-the-art results among open-source 7B MLLMs on multiple challenging benchmarks, including:
MathVerse (+5.4%)
MathVision (+13.9%)
DynaMath (+9.8%)
WeMath (+0.2%)
LogicVista (+9.6%)
AIME24 (+44.6%)
AIME25 (+15.4%)
GPQA (+10.1%)
MATH500 (+23.4%)
These improvements ranged from 4.5% to 44.6% over previous models, demonstrating significant gains in multimodal reasoning.
Ablation studies confirmed the importance of each training stage and the PAD mechanism, with the full three-stage curriculum outperforming variants by an average of 16.8%. The Efficient-Length Reward contributed to training stability and response conciseness. Limitations include the focus on 7B parameter models and the need to explore scalability to larger architectures and diverse modalities.
Why does this matter?
This work advances the field of multimodal AI by showing that a carefully designed, staged training curriculum combined with innovative reinforcement learning techniques can substantially improve reasoning across language and vision modalities. The open-source release of ReVisual-R1 enables researchers and practitioners to build more capable, generalizable multimodal AI systems that can better understand and reason about complex, real-world data involving text, images, and beyond. Such improvements have broad implications for applications like intelligent assistants, educational tools, and AI-powered analysis of multimedia content, pushing the boundaries of how machines integrate and reason over multiple modalities.
Key Points
Introduces ReVisual-R1, a 7B open-source multimodal large language model with enhanced reasoning capabilities.
Employs a three-stage training curriculum: cold start with text, multimodal reinforcement learning with GRPO and PAD, and text-only reinforcement learning.
Achieves state-of-the-art performance on multiple multimodal reasoning benchmarks, with significant improvements over prior models.
Demonstrates the effectiveness of staged training and novel reinforcement learning techniques in advancing multimodal reasoning.
Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
What’s the research question?
How well do Multimodal Large Language Models (MLLMs) perform in structured visual reasoning tasks involving multiple images?
What did the authors do?
The authors developed a comprehensive benchmark called the Multimodal Multi-image Reasoning Benchmark (MMRB) to evaluate MLLMs on complex reasoning tasks across multiple images. Their approach included:
Designing 92 sub-tasks covering spatial, temporal, and semantic reasoning involving multiple images.
Annotating reasoning paths using GPT-4o and refining them with expert human corrections to ensure high-quality ground truth.
Creating a subset of tasks specifically for evaluating multimodal reward models.
Developing a sentence-level matching framework using open-source LLMs to enable scalable evaluation of large model outputs.
Evaluating 40 MLLMs, including 9 reasoning-specific models and 8 reward models, on the MMRB benchmark.
What did they find?
The study revealed several key insights:
Open-source MLLMs significantly lag behind commercial models in multi-image reasoning, with average scores below 50%.
Models with larger size and those employing explicit Chain-of-Thought prompting performed better, highlighting the importance of model scale and reasoning strategies.
Multimodal reward models exhibited instability in multi-image reward tasks, indicating challenges in reliably guiding model improvements through reward signals.
Why does this matter?
This work provides a critical step forward in evaluating and understanding the capabilities of MLLMs in complex, real-world scenarios involving multiple images. By establishing a detailed and challenging benchmark, it highlights the current limitations of open-source models and multimodal reward systems, encouraging the AI community to develop more robust and capable multi-image reasoning architectures. Improving these models can enhance applications in areas such as visual question answering, multimedia analysis, and autonomous agents that need to interpret and reason over rich visual data from multiple sources.
Key Points
Introduces MMRB, a comprehensive benchmark for structured reasoning over multiple images.
Shows open-source MLLMs underperform compared to commercial counterparts in multi-image tasks.
Highlights the benefits of larger models and explicit Chain-of-Thought prompting for reasoning.
Identifies instability issues in multimodal reward models, pointing to future research directions.
Multimodal Tabular Reasoning with Privileged Structured Information
Image from arXiv paper.
What’s the research question?
How can privileged structured information be leveraged during training to improve multimodal large language models (MLLMs) in tabular reasoning tasks involving table images?
What did the authors do?
The authors introduced the Turbo framework, a novel training approach designed to enhance MLLMs' reasoning over table images by using structured tables as privileged information during training. The key components include:
Two-stage training process: combining supervised fine-tuning (SFT) and reinforcement learning (RL).
SFT stage: generating high-quality reasoning traces with DeepSeek-R1, a state-of-the-art reasoning language model, which creates [question, reasoning, answer] triples emphasizing question-relevant reasoning and filtering out redundant content.
Think-then-answer paradigm: training the model to produce coherent reasoning chains before generating final answers.
RL stage: generating multiple candidate answers per question, evaluating their relative advantages via reward comparisons, and updating the model's policy using Group Relative Policy Optimization (GRPO) to favor better responses.
What did they find?
The Turbo framework achieved state-of-the-art performance with a +7.2% improvement across multiple datasets, especially excelling in complex reasoning tasks. Ablation studies confirmed the importance of both training stages, showing that the combination of structure-aware reasoning trace generation and iterative policy optimization significantly boosts reasoning fidelity and answer consistency. Case studies demonstrated Turbo’s ability to handle diverse reasoning challenges, including:
Visual grounding in table images
Mathematical reasoning
Logical inference
Limitations include potential dependence on the quality of reasoning traces and the need for high-quality structured table data during training.Why does this matter?
This work advances the field of multimodal reasoning by showing how structured tabular data can serve as privileged information to guide training, leading to more interpretable and robust MLLMs. The Turbo framework’s ability to produce coherent reasoning chains and improve performance on complex tasks has broad implications for real-world applications in domains like finance, healthcare, and scientific research, where understanding and reasoning over multimodal data is critical. By integrating structured reasoning traces with reinforcement learning, Turbo sets a new standard for systematic and interpretable multimodal reasoning models.
Key Points
Introduces Turbo, a two-stage training framework combining supervised fine-tuning and reinforcement learning for multimodal tabular reasoning.
Leverages structured tables as privileged information during training to improve reasoning over table images.
Produces high-quality reasoning traces that enhance interpretability and reasoning fidelity.
Achieves state-of-the-art results (+7.2%) on multiple datasets, excelling in complex reasoning tasks.
Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning
What’s the research question?
Can smaller, carefully selected high-value datasets match or outperform full datasets in training multi-modal large language models (MLLMs) for complex reasoning tasks involving text and images?
What did the authors do?
The authors introduced a novel approach to identify the most valuable training samples—called cognitive samples—that effectively stimulate multi-modal reasoning in MLLMs. Their methodology includes:
Reasoning Activation Potential (RAP) paradigm: A framework to estimate how well each training sample activates reasoning capabilities in the model.
Output-level reasoning discrepancy: Uses the Potential Outcome Model (POM) to compare the model’s predictions when given combined text and image inputs versus text-only inputs, measuring reliance on visual information.
Process-level reasoning confidence: Employs the Attention Confidence Estimator (ACE) to analyze token-level self-attention patterns, identifying samples where the model over-attends to irrelevant tokens.
Difficulty-aware Replacement Module (DRM): Ensures training data maintains sufficient complexity by replacing trivial samples with cognitively challenging ones.
What did they find?
The RAP-based data selection method demonstrated remarkable efficiency and effectiveness:
Achieved superior multi-modal reasoning performance using only 9.3% of the full training data.
Reduced computational costs by over 43%.
Outperformed models trained on the entire dataset across six different reasoning benchmarks.
Showed that a small, high-value subset of data can drive better reasoning than larger, less targeted datasets.
Limitations include potential challenges in generalizing the cognitive sample selection to other modalities or tasks without further adaptation.
Why does this matter?
This work challenges the common assumption that bigger datasets always lead to better model performance. By intelligently selecting high-value cognitive samples, it offers a more efficient way to train powerful multi-modal models that can reason across text and images. This approach can significantly reduce training costs and computational resources, making advanced multi-modal reasoning more accessible and scalable. It also opens new avenues for developing models that focus on truly informative data, potentially improving their ability to generalize and reason in real-world applications such as visual question answering, multimedia search, and human-AI interaction.
Key Points
Introduces RAP paradigm to identify samples that best activate multi-modal reasoning in MLLMs.
Uses output-level discrepancy and process-level attention confidence to select high-value data.
Achieves high performance with less than 10% of the full training data, reducing costs significantly.
Demonstrates that targeted data selection can outperform full dataset training for complex reasoning tasks.
Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
Image from arXiv paper.
What’s the research question?
How does reinforcement learning (RL) influence the reasoning capabilities of large language models (LLMs) across different components of mathematical reasoning?
What did the authors do?
The authors developed a comprehensive framework called SPARKLE to analyze how RL fine-tuning affects LLMs' reasoning skills, focusing on three key components:
Plan-following and execution: How well models follow and implement reasoning plans.
Knowledge utilization: How effectively models incorporate relevant external knowledge.
Subproblem decomposition: How models break down complex problems into manageable parts.
They created the SPARKLE benchmark, which enhances mathematical datasets with:
Planning skeletons to guide reasoning steps.
Relevant external knowledge snippets.
Decompositions of problems into subproblems.
The study compares LLMs before and after RL fine-tuning using this framework, evaluating how RL impacts each reasoning component, especially on difficult and knowledge-intensive problems.
What did they find?
RL fine-tuning improves certain reasoning aspects but not others:
Strengths: Enhanced plan-following accuracy and better integration of external knowledge, leading to improved performance on complex problems.
Weaknesses: Continued challenges in detailed subproblem resolution, with models struggling to fully decompose and solve multi-step subproblems.
Additional insights: Multi-stage RL approaches with partial solution augmentation further boost reasoning performance.
While RL helps models better follow reasoning plans and leverage external info, it does not fully address the complexity of breaking down and solving intricate subproblems.
Why does this matter?
This work offers a nuanced understanding of how RL shapes different facets of mathematical reasoning in LLMs, highlighting strengths and limitations. The SPARKLE framework and benchmark enable researchers to dissect reasoning behaviors in detail, guiding the design of more capable, interpretable, and robust reasoning models. By pinpointing which reasoning components benefit from RL and which need further improvement, this study helps accelerate the development of LLMs that can better understand, plan, and solve complex mathematical problems, with potential applications in education, scientific research, and automated reasoning systems.
Key Points
Introduces SPARKLE, a three-axis framework for analyzing LLM reasoning components: plan-following, knowledge use, and subproblem decomposition.
Uses an augmented mathematical benchmark to evaluate how RL fine-tuning affects reasoning performance.
Finds RL improves plan-following and knowledge integration but struggles with detailed subproblem resolution.
Highlights the importance of targeted training strategies to enhance all reasoning facets in LLMs.
Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
What’s the research question?
How can we rigorously evaluate and improve the logical reasoning capabilities of large language models (LLMs)?
What did the authors do?
The authors developed FineLogic, a comprehensive framework to assess and enhance the logical reasoning of LLMs through fine-grained analysis and supervision strategies. Their approach includes:
Evaluating models on four diverse datasets (FLD, FOLIO, Multi-LogiEval, ProntoQA) that require multiple reasoning steps with varying formats and depths.
Measuring overall accuracy (final answer correctness), stepwise soundness (validity, relevance, atomicity of each reasoning step), and internal representation alignment (how well internal states reflect logical structure).
Using GPT-4.1-mini to validate intermediate reasoning steps for soundness.
Comparing two supervision styles during fine-tuning: natural language supervision (NL) and symbolic reasoning (Structured, Filtered, Direct).
What did they find?
The study revealed distinct strengths and weaknesses of the supervision strategies:
NL-supervised models (e.g., Llama-3.1-SFT-NL) achieved high overall accuracy (e.g., 89.5% on FLD, 93.5% on ProntoQA) and excelled in out-of-distribution generalization.
Symbolic models (e.g., Llama-3.1-SFT-Symb-Filter) produced more atomic, rule-based reasoning chains, with better stepwise soundness (e.g., 35.0% all valid steps) and fact relevance.
NL models struggled with minimal, rule-aligned reasoning but offered better interpretability at the representation level.
Symbolic models provided more logically grounded inference chains but lagged in overall accuracy and generalization.
Combining both supervision styles could potentially leverage the strengths of each: NL’s flexibility and symbolic reasoning’s structure.
Why does this matter?
This work advances our understanding of how to evaluate and improve the logical reasoning of LLMs in a detailed and interpretable manner. By dissecting reasoning into accuracy, step validity, and internal structure, FineLogic offers a nuanced lens that can guide the development of more robust, transparent, and generalizable reasoning systems. Such improvements are crucial for deploying LLMs in applications requiring complex, multi-step inference, such as scientific reasoning, legal analysis, and advanced question answering. The study also highlights the potential of hybrid supervision strategies to balance flexibility and structure, paving the way for next-generation AI systems capable of more reliable and explainable reasoning.
Key Points
Introduces FineLogic, a detailed evaluation framework for LLM logical reasoning.
Assesses models on accuracy, stepwise soundness, and internal representation alignment.
Finds natural language supervision excels in generalization, while symbolic styles improve reasoning structure.
Suggests combining supervision strategies to optimize reasoning capabilities.
Reason-to-Recommend: Using Interaction-of-Thought Reasoning to Enhance LLM Recommendation
Image from arXiv paper.
What’s the research question?
Can structured reasoning over user-item interactions improve the recommendation capabilities of large language models (LLMs)?
What did the authors do?
The authors introduced the R2Rec framework, which enhances LLM-based recommendations by integrating structured reasoning about user-item interactions through the following steps:
Interaction chains construction: They build interaction graphs from user-item data and convert these into sequential reasoning chains called Interaction-of-Thoughts.
Progressive prompting: A masked prompting strategy guides the LLM to generate reasoning steps grounded in the interaction context.
Two-stage training: The model undergoes supervised fine-tuning (SFT) on high-quality annotated reasoning chains, followed by reinforcement learning (RL) to further optimize reasoning quality and recommendation accuracy using reward signals.
What did they find?
The R2Rec approach achieved significant improvements:
Average 10.48% better HitRatio@1 compared to baseline models across three datasets.
Outperformed classical graph-based models (e.g., NGCF, LightGCN) and other LLM-based recommendation methods (e.g., ChatRec, LLaRA).
Ablation studies confirmed that both the interaction chains and the two-stage training process are crucial for performance gains.
Transferability tests showed that the reasoning approach generalizes well across different domains.
Limitations include the reliance on high-quality annotated reasoning data and potential computational complexity of reasoning chains.
Why does this matter?
This work demonstrates that embedding structured, step-by-step reasoning about user preferences and item interactions into LLMs can significantly boost recommendation accuracy and transparency. By making the reasoning process explicit, R2Rec not only improves predictions but also offers interpretable insights into why certain items are recommended. This advancement bridges the gap between powerful language models and effective recommender systems, paving the way for more intelligent, reasoning-aware personalized recommendations that can adapt across diverse domains and user behaviors.
Key Points
Introduces Interaction-of-Thoughts to encode user-item interaction chains for LLM recommendation.
Combines supervised fine-tuning and reinforcement learning to optimize reasoning and recommendation quality.
Achieves over 10% improvement in top-1 recommendation accuracy compared to baselines.
Enhances interpretability by making the reasoning process explicit and grounded in interaction data.
Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation
What’s the research question?
How can reinforcement learning be used to improve the integration of parametric and contextual knowledge in large language models?
What did the authors do?
The authors introduced Knowledgeable-r1, a novel reinforcement learning framework designed to enhance how large language models (LLMs) combine two types of knowledge:
Parametric knowledge: stored within the model’s weights.
Contextual knowledge: retrieved from external sources during inference.
The framework features three main components:
Knowledge capability exploration: Jointly probes parametric and contextual knowledge by sampling both simultaneously, evaluating their usefulness in real-time.
Knowledge capability optimization: Uses three separate policies—one for parametric knowledge, one for contextual knowledge, and one combining both—to learn how best to leverage each source.
Knowledge advantage adjustment: Applies a novel advantage function transformation T(A) to dynamically weight the importance of each knowledge pathway based on the current task context.
Training involves policy gradient methods that update these policies iteratively, guided by advantage functions and a regularization term based on KL divergence to ensure stability.
What did they find?
Knowledgeable-r1 achieved significant improvements over baseline retrieval-augmented generation (RAG) prompting:
On the ConflictQA benchmark, it reached an average accuracy of 68.27%, outperforming the baseline by 17.07%.
Demonstrated robustness to extraneous knowledge with gains of 7.5% and 5.45% in specific metrics (Acc SCTI and Acc SCFI).
Improved knowledge fusion, evidenced by a 5.4% increase in the Acc TiTe metric.
Enhanced performance in knowledge extension tasks with a 6.7% accuracy boost.
Consistent improvements across in-domain and out-of-domain tasks, even when both knowledge sources were correct.
Limitations include the need for careful tuning of the advantage function transformation and potential computational overhead due to multiple policy components.
Why does this matter?
This work advances the ability of large language models to effectively combine their internal (parametric) knowledge with external (contextual) information, a key challenge in deploying robust and intelligent AI systems in real-world scenarios. By enabling more dynamic and optimized exploration of both knowledge sources, Knowledgeable-r1 enhances reasoning accuracy and model robustness, making LLMs more reliable for applications like fact-checking, question answering, and dialogue systems. Its modular reinforcement learning approach opens new research directions in knowledge fusion and exploration, contributing to the development of more adaptable and context-aware AI agents.
Key Points
Introduces Knowledgeable-r1, a reinforcement learning framework for knowledge integration in LLMs.
Jointly explores and optimizes parametric and contextual knowledge pathways.
Uses a novel advantage function transformation to dynamically weight knowledge sources based on context.
Achieves significant accuracy improvements on knowledge conflict and extension benchmarks.
LLMs for sensory-motor control: Combining in-context and iterative learning
What’s the research question?
Can large language models (LLMs) effectively control embodied agents by directly mapping continuous observation vectors to continuous action vectors without relying on predefined motor primitives?
What did the authors do?
The authors developed a novel approach to using LLMs for controlling physical agents that interact with their environment. Their methodology includes:
Structured prompting: They designed prompts that guide LLMs to generate control policies based on sensory inputs and goals, without requiring large demonstration datasets or predefined motor commands.
Two-phase control strategy: - Initial policy generation: The LLM receives a textual description of the agent, environment, and objectives, and outputs a high-level control policy expressed as IF-THEN-ELSE rules.
- Iterative refinement: The LLM repeatedly revises its control policy by incorporating performance feedback and sensory-motor data collected during agent evaluation.Policy translation: The generated rules are translated into executable Python code to control the agent in simulation environments.
What did they find?
The proposed method was tested on classic control tasks from the Gymnasium library and the inverted pendulum task from MuJoCo. Key findings include:
The largest model tested, Qwen2.5:72B, achieved the highest average reward (~350) on CartPole tasks, outperforming baseline approaches.
The iterative learning process significantly improved control performance compared to the initial policies generated by the LLM.
The approach demonstrated robustness across different tasks, successfully adapting control strategies without predefined motor primitives.
Limitations include reliance on simulation environments and the need for careful prompt design; real-world transfer remains to be explored.
Why does this matter?
This work challenges the traditional reliance on predefined motor primitives and large demonstration datasets for controlling embodied agents. By leveraging LLMs' symbolic reasoning and language understanding capabilities, combined with structured prompting and iterative refinement, the authors show that LLMs can directly generate and improve continuous control policies. This opens exciting new possibilities for autonomous agents and embodied AI, enabling more flexible, adaptable, and data-efficient control strategies that integrate sensory-motor data with high-level reasoning. Such advances could impact robotics, simulation-based training, and human-AI interaction by providing a powerful, generalizable approach to sensory-motor control.
Key Points
Introduces a structured prompting method for LLMs to generate control policies from sensory-motor data.
Combines symbolic rule-based control with iterative refinement based on performance feedback.
Achieves strong results on classic control benchmarks without predefined motor primitives.
Demonstrates potential for LLMs to control embodied agents directly, opening new avenues for autonomous systems.
Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
What’s the research question?
Can combining multi-agent debate with structured preference optimization improve the training efficiency and reasoning capabilities of smaller language models?
What did the authors do?
The authors introduced a novel framework called Debate and Reflect (D&R) that enhances small language models through collaborative debate and structured feedback:
Multi-Agent Debates: Small student models engage in multi-turn debates with stronger teacher models, generating diverse responses and self-reflections.
Structured Interaction Graphs (MAGs): Debate interactions are recorded in Multi-Agent Interaction Graphs, capturing responses, reflections, and teacher feedback for later analysis.
Tree-structured Direct Preference Optimization (T-DPO): Responses are organized into hierarchical preference trees where correct answers are linked as 'chosen' nodes and incorrect ones as 'rejected'.
Training Stages: The framework includes Supervised Fine-Tuning (SFT) on gold-standard data and T-DPO fine-tuning on the structured debate logs, enabling the model to learn both preferred responses and underlying reasoning.
What did they find?
The D&R framework led to significant improvements in small language model performance:
Enhanced Reasoning: Achieved an average accuracy of 38.16 on the MMLU Pro benchmark, outperforming baseline distillation methods.
Improved Math Reasoning: Increased MATH dataset score from 8.02 to 17.32, demonstrating better multi-step problem-solving.
Efficiency Gains: Reduced token costs during inference and improved generalization across diverse tasks.
Self-Correction: Learned to identify and correct errors during inference thanks to structured debate feedback.
Ablation Insights: Both self-reflection and teacher feedback were crucial; removing either degraded performance.
Robustness: Effective across different data scales and model sizes, indicating broad applicability.
Why does this matter?
This work offers a scalable and effective approach to boosting small language models by leveraging multi-agent debate and structured preference learning. By enabling models to internalize reasoning strategies and learn from hierarchical feedback, D&R enhances reasoning, robustness, and efficiency—key qualities for deploying AI in resource-constrained environments. Its emphasis on structured interactions and self-reflection could influence future research on model alignment, interpretability, and collaborative AI systems, paving the way for smarter, more reliable language models that can reason like humans while remaining computationally efficient.
Key Points
Introduces Debate and Reflect (D&R), a framework combining multi-agent debate with structured preference optimization.
Uses Tree-structured Direct Preference Optimization (T-DPO) to organize debate responses into hierarchical preference trees.
Achieves state-of-the-art improvements in reasoning benchmarks for small language models.
Enhances model efficiency and robustness through structured feedback and self-reflection.