Developing effective multi-modal AI systems for real-world applications requires handling diverse tasks such as fine-grained recognition, visual grounding, reasoning, and multi-step problem-solving. Existing open-source multi-modal language models are found to be wanting in these areas, especially for tasks that involve external tools such as OCR or mathematical calculations. The abovementioned limitations can largely be attributed to single-step-oriented datasets that cannot provide a coherent framework for multiple steps of reasoning and logical chains of actions. Overcoming these will be indispensable for unlocking true potential in using multi-modal AI on complex levels.
Current multi-modal models often rely on instruction tuning with direct-answer datasets or few-shot prompting approaches. The proprietary systems, like GPT-4, have demonstrated the ability to reason through CoTA chains effectively. At the same time, the open-source models face challenges from a lack of datasets and integration with tools. The earlier efforts, such as LLaVa-Plus and Visual Program Distillation, were also limited by small dataset sizes, poor-quality training data, and a focus on simple question-answering tasks, which limits them to more complex multi-modal problems that require more sophisticated reasoning and tool application.
Researchers from the University of Washington and Salesforce Research have developed TACO, an innovative framework for training multi-modal action models using synthetic CoTA datasets. This work introduces several key advancements to address the limitations of prior methods. First, over 1.8 million traces have been generated using GPT-4 and programs in Python, while a subset of 293K examples was curated to present high quality after rigorous filtering techniques. These examples ensure the inclusion of diverse reasoning and action sequences critical for multi-modal learning. Second, TACO incorporates a robust set of 15 tools, including OCR, object localization, and mathematical solvers, enabling the models to handle complex tasks effectively. Third, advanced filtering and data-mixing techniques further optimize the dataset, emphasizing reasoning-action integration and fostering superior learning outcomes. This framework reinterprets multi-modal learning by enabling models to produce coherent multi-step reasoning while seamlessly integrating actions, thereby establishing a new benchmark for performance in intricate scenarios.
The development of TACO involved training on a carefully curated CoTA dataset with 293K instances sourced from 31 different origins, including Visual Genome. These datasets contain a wide range of tasks such as mathematical reasoning, optical character recognition, and detailed visual understanding. It is highly heterogeneous, with the tools provided including object localization and language-based solvers that allow a wide range of reasoning and action tasks. The training architecture combined LLaMA3 as the linguistic basis with CLIP as the visual encoder thus establishing a strong multi-modal framework. Fine-tuning established hyperparameter tuning that focused on lowering learning rates and increasing the number of epochs for training to guarantee that the models could adequately solve complex multi-modal challenges.
TACO’s performance across eight benchmarks demonstrates its substantial impact on advancing multi-modal reasoning capabilities. The system achieved an average accuracy improvement of 3.6% over instruction-tuned baselines, with gains as high as 15% on MMVet tasks that involve OCR and mathematical reasoning. Notably, the high-quality 293K CoTA dataset outperformed larger, less refined datasets, underscoring the importance of targeted data curation. Further performance improvements are achieved through adjustments in hyperparameter strategies, including tuning of vision encoders and optimization of learning rates. Table 2: Results show excellent performances by TACO compared to benchmarks, the latter was found to be exceptionally better in complex tasks involving integration of reasoning and action.
TACO introduces a new methodology for multi-modal action modeling that effectively addresses the severe deficiencies of both reasoning and tool-based actions through high-quality synthetic datasets and innovative training methodologies. The research overcomes the constraints of traditional instruction-tuned models, and its developments are poised to change the face of real-world applications ranging from visual question answering to complex multi-step reasoning tasks.
Check out the Paper, GitHub Page, and Project Page. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 65k+ ML SubReddit.
🚨 FREE UPCOMING AI WEBINAR (JAN 15, 2025): Boost LLM Accuracy with Synthetic Data and Evaluation Intelligence–Join this webinar to gain actionable insights into boosting LLM model performance and accuracy while safeguarding data privacy.
Aswin AK is a consulting intern at MarkTechPost. He is pursuing his Dual Degree at the Indian Institute of Technology, Kharagpur. He is passionate about data science and machine learning, bringing a strong academic background and hands-on experience in solving real-life cross-domain challenges.
