Exploring Multimodal Prompt for Visualization Authoring with Large Language Models
Zhen Wen1
Luoxuan Weng1
Yinghao Tang1
Runjin Zhang1
Yuxin Liu1
Bo Pan1
Minfeng Zhu2
Wei Chen1
1State Key Lab of CAD&CG, Zhejiang University 2Zhejiang University
Multimodal prompt for visualization authoring with VisPilot. (A) The user can create visualizations by providing sketching, text annotations or directly manipulating existing visualizations. (B) VisPilot interprets the multimodal input and generates visualizations.
Abstract
Recent advances in large language models (LLMs) have shown great potential in automating the process of visualization authoring through simple natural language utterances. However, instructing LLMs using natural language is limited in precision and expressiveness for conveying visualization intent, leading to misinterpretation and time-consuming iterations. To address these limitations, we conduct an empirical study to understand how LLMs interpret ambiguous or incomplete text prompts in the context of visualization authoring, and the conditions making LLMs misinterpret user intent. Informed by the findings, we introduce visual prompts as a complementary input modality to text prompts, which help clarify user intent and improve LLMs' interpretation abilities. To explore the potential of multimodal prompting in visualization authoring, we design VisPilot, which enables users to easily create visualizations using multimodal prompts, including text, sketches, and direct manipulations on existing visualizations. We evaluate VisPilot through a controlled user study and an expert evaluation. The results suggest that multimodal prompts facilitate users in communicating spatial constraints, local references, and design preferences while maintaining comparable task efficiency to text-only prompting. We further discuss when text, visual, and hybrid prompts are beneficial for visualization authoring, and summarize design implications for future human-AI authoring systems. All materials are available at https://osf.io/2qrak.
The VisPilot System
The interface of VisPilot includes four components: (A) Chat Interface, (B) Free-drawing Canvas, (C) Design Panel, and (D) Authoring Flow.
Multimodal Prompt Framework
Our prompting framework instructs the LLM to interpret visual prompts to visualization specifications step by step.
Use Cases
Case 1 - Visualization Authoring
The video demonstrates the process of creating a sophisticated visualization using VisPilot (full video coming soon).
Case 2 - Data Exploration
Case study with domain experts analyzing Premier League player statistics, using VisPilot to iteratively create visualizations from (A) to (D). View full-size image.
Research Contributions
- Empirical study on LLMs' interpretation of natural language prompts for visualization authoring
- Novel prompting framework for multimodal visualization authoring
- Interactive system for visualization authoring
- User study findings on multimodal visualization requests
Citation
@article{wen2026exploring,
title = {Exploring Multimodal Prompt for Visualization Authoring with Large Language Models},
author = {Zhen Wen and Luoxuan Weng and Yinghao Tang and Runjin Zhang and Yuxin Liu and Bo Pan and Minfeng Zhu and Wei Chen},
journal = {IEEE Transactions on Visualization and Computer Graphics},
year = {2026},
volume = {32},
number = {9},
pages = {7685--7700},
doi = {10.1109/TVCG.2026.3701510}
}