ChartQA
ChartQA tests question answering over charts, requiring visual data extraction plus arithmetic and comparison reasoning. It uses relaxed numeric accuracy across human-authored and machine-augmented question sets, with the human set being the harder, more telling measure.
ChartQA evaluates whether a model can answer questions about data visualizations such as bar charts, line graphs, and pie charts. Charts encode numbers visually, so answering often requires both reading values off the chart and performing arithmetic or comparison over them, a combination of perception and reasoning.
What It Measures
Questions range from simple lookups ("What is the value for 2021?") to compositional reasoning ("How much larger is the highest bar than the lowest?" or "What is the average across all years?"). The model must extract the right data points from the visual, then compute or compare. ChartQA therefore tests visual data extraction, numerical reasoning, and the ability to ground a question in the correct chart elements such as axes, legends, and labels.
Methodology
ChartQA includes two question sets. A human-authored set contains natural questions written by people, which tend to be reasoning-heavy. A machine-augmented set is generated automatically from chart data tables to broaden coverage. The standard metric is relaxed accuracy, which counts a numeric answer correct if it falls within a small tolerance of the ground truth, acknowledging that reading values off a chart introduces minor imprecision. Scores are reported separately for the human and augmented sets and averaged.
How to Interpret Results
Compare human-set and augmented-set accuracy. The human set is harder and more reasoning-intensive, so it better reflects real analytical ability, while the augmented set checks breadth. A model strong on lookups but weak on the human set can read charts but not reason over them. The relaxed-accuracy tolerance matters: report it, since a looser tolerance inflates scores. For business-analytics use cases, the human-set number is the one to weigh most.
Limitations
Relaxed numeric tolerance can credit answers that are close but practically wrong, and choosing the tolerance is a judgment call. Chart styles in the dataset may not match the cluttered or custom visuals seen in practice, limiting transfer. The benchmark focuses on common chart types and short answers, under-testing complex dashboards or multi-chart reasoning. As with other vision-language benchmarks, popular splits saturate over time and face contamination risk.