You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Mini-o3 is an open-source model that delivers strong thinking-with-images capability, generating multi-turn agentic trajectories like OpenAI o3, with the interaction turns <spanstyle="color: #ff3860">scaling up to tens of rounds</span>.
210
-
Full training recipe is open-source.
204
+
AdaptVision is an open-source model that leverages agentic visual tool use for dynamic visual token reduction, achieving a sota-level accuracy-efficiency trade-off across multiple VQA benchmarks.
Recent advances in large multimodal models have leveraged image-based tools with reinforcement learning to tackle visual problems. However, existing open-source approaches often exhibit monotonous reasoning patterns and allow only a limited number of interaction turns, making them inadequate for difficult tasks that require trial-and-error exploration. In this work, we address this limitation by <b>scaling up tool-based interactions</b> and introduce <b>Mini-o3</b>, a system that executes deep, multi-turn reasoning—spanning tens of steps—and achieves state-of-the-art performance on challenging visual search tasks. Our full recipe for reproducing OpenAI o3–style behaviors is presented.
222
+
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
229
223
<oltype="1">
230
-
<li><b>Challenging Dataset</b>. <spanstyle="font-size: 95%;">We construct the <spanstyle="color: #ff3860">Visual Probe Dataset</span>, a collection of thousands of challenging visual search problems designed for exploratory reasoning.</span></li>
231
-
<li><b>Diverse Multi-turn Trajectories for Cold-start</b>. <spanstyle="font-size: 95%;">We develop an iterative data collection pipeline to obtain cold-start trajectories that exhibit <spanstyle="color: #ff3860">diverse reasoning patterns, including depth-first search, trial-and-error, and goal maintenance</span>.</li>
224
+
<li><b>Synergizing Visual Reasoning and Visual Token Compression</b>. <spanstyle="font-size: 95%;">We introduce AdaptVision, a VLM framework that leverages visual tool use for dynamic token reduction.</span></li>
225
+
<li><b>Efficient Algorithm</b>. <spanstyle="font-size: 95%;">We propose a Decoupled Turn Policy Optimization (DTPO) algorithm alongside a tailored reward function to enable the effective training of AdaptVision.</span></li>
226
+
<li><b>Performance</b>. <spanstyle="font-size: 95%;">Extensive evaluation on multiple VQA benchmarks shows that AdaptVision achieves superior performance with substantially reduced visual token consumption compared to existing efficient VLM methods.</span></li>
227
+
<li><b>Open-source</b>. <spanstyle="font-size: 95%;">All code, models, and datasets are available to facilitate reproducibility and further research.</span></li>
228
+
<!-- <li><b>Challenging Dataset</b>. <span style="font-size: 95%;">We construct the <span style="color: #ff3860">Visual Probe Dataset</span>, a collection of thousands of challenging visual search problems designed for exploratory reasoning.</span></li> -->
229
+
<!-- <li><b>Diverse Multi-turn Trajectories for Cold-start</b>. <span style="font-size: 95%;">We develop an iterative data collection pipeline to obtain cold-start trajectories that exhibit <span style="color: #ff3860">diverse reasoning patterns, including depth-first search, trial-and-error, and goal maintenance</span>.</li>
232
230
<li><b>Test-time Turns Scaling</b>. <span style="font-size: 95%;">We propose an <b>over-turn masking</b> strategy that prevents penalization of responses exceeding the maximum turns during reinforcement learning, thereby balancing training-time efficiency with test-time scalability. Despite training with an upper bound of only six interaction turns, our model generates trajectories that naturally <span style="color: #ff3860">scale to tens of turns at inference time, with accuracy improving as the number of turns increases</span>.</li>
233
231
<li><b>Performance</b>. <span style="font-size: 95%;">Extensive experiments demonstrate that Mini-o3 produces rich reasoning patterns and deep thinking paths, effectively solving challenging visual problems, thereby achieving the <span style="color: #ff3860">state-of-the-art results on a variety of benchmarks</span> (e.g., VisualProbe, V* Bench, HR-Bench, MME-Realworld).</li>
234
-
<li><b>Open-source</b>. <spanstyle="font-size: 95%;">All code, models, and datasets are available to facilitate reproducibility and further research.</li>
232
+
<li><b>Open-source</b>. <span style="font-size: 95%;">All code, models, and datasets are available to facilitate reproducibility and further research.</li> -->
0 commit comments