Can LLMs Locate and Correct Erroneous Tokens?
Panda-CVL: A Vision-Language Dataset for Token-Level Correction

^{1          2}

The token-level correction interface used for annotating Panda-CVL
For details about the annotation tool, see onPanda

Abstract

We release Panda-CVL, a dataset and accompanying benchmark for evaluating models' token-level correction capability: given a question-response pair, the model must first judge whether the response is acceptable; if it decides a correction is needed, it must locate the first inappropriate token in the response and correct it to an appropriate one--so that the policy model can continue generating from the "correct prefix + corrected token" and ultimately produce an acceptable response. Compared with prior data that only identifies which step an error occurs in, Panda-CVL brings two improvements: (1) it refines error localization to the token level; (2) the model must not only point out which token is inappropriate but also provide the correct token it should be changed to, yielding supervision signals that are precise in both position and correction direction. Panda-CVL was annotated by human annotators using the onPanda annotation tool and contains 7,491 annotation sessions, split into a training set of 6,839 and a test set of 652; the training set can be directly used to strengthen models' token-level correction capability. Panda-CVL is annotated mainly on Chinese vision-language data; we additionally provide an English-only subset, Panda-MultiRef-21, which can be used to separately evaluate models' token-level correction capability in English. Evaluation of recent LLMs shows that this task remains challenging--the best model achieves an F1 of only 17.09. We further analyze the inter-annotator consistency of token-level correction through a controlled experiment, providing public resources and empirical references for research on this new type of data.