Generative AI systems can solve math problems in seconds, but for students learning math, working through a problem matters far more than simply getting it right. Yet AI still struggles to evaluate student reasoning and provide the kind of feedback a teacher gives to help students take the next step without doing the thinking for them.
A team of Portland State University researchers is helping bridge that gap through a new project — "Seeing Through Data: Benchmarking Adaptive Feedback on Grades 6–12 Data Reasoning" — that will develop evaluation tools and benchmarks for testing how well AI systems respond to students’ mathematical reasoning.
Eva Thanheiser, a professor of mathematics education, and Ameeta Agrawal, an associate professor of computer science, are evaluating whether AI models can recognize the mathematical reasoning in a student's response and suggest a useful next step. They are joined by Amanda Sugimoto, professor of education, who leads the project’s literacy component.
But they're adamant about one thing: It's not about replacing teachers; it's about creating better ways to evaluate AI that could eventually be used to support teaching and learning without compromising authentic learning or student privacy.
The two-year project is supported by a $2 million grant from the Gates Foundation and Learning Commons through the Digital Promise K-12 AI Infrastructure Program. The initiative focuses on developing openly shared datasets, models and benchmarks to advance the accuracy and relevance of AI in education.
"Any model can give you feedback, but there are degrees of feedback," Agrawal said. "No matter the system, we want to give different dimensions and evaluate how well the feedback responds to student reasoning. How well does Claude give scaffolding versus blurting out the answer? How well does GPT-6 Astra improve a student's reasoning versus something else?"
The dimensions that the models will be scored against will be informed by real math tasks collected from students and teachers in partnering school districts.
Thanheiser will lead that effort, working with teachers to develop tasks that invite multiple forms of reasoning. Teachers will then guide their students in a back-and-forth loop, identifying the student's reasoning, suggesting next steps and repeating as students respond.
"The feedback is grounded in the student's thinking rather than in the correct answer," she said.
The tasks will focus on data visualization reasoning, which Thanheiser calls one of the most important forms of communication today.
"To participate in larger conversations, you have to figure out what the intent was behind the data visualization so you can consume it properly," she said. "We know this is really difficult for a lot of students."
Thanheiser said critical data literacy is not simply about students making graphs but about interpreting, critiquing and using data to make sense of real-world issues. Students have to decide what data to include, how to represent those data, what comparisons are fair, what claims are warranted and what additional information is needed.
And students encounter graphs beyond math class. They appear in science articles, in the news and on their phones, usually attached to a claim somebody wants them to believe. Sugimoto is extending the work into English language arts, building tasks around infographics and data-based claims in informational text. Her research focuses on how linguistically and culturally diverse students are assessed, which, for this project, means asking whether AI feedback recognizes sound reasoning when it doesn't arrive in polished English. Recent evaluations suggest this is where these systems are weakest, performing worst on work from the students who need the most support.
Thanheiser said these types of data visualization problems serve as a good test case for evaluating whether AI-generated feedback can recognize and respond to students' reasoning.
For example, a student might create a bar graph showing that School A has 80 suspensions and School B has 50 suspensions, then claim that School A has a "bigger discipline problem." Thanheiser said a weak AI response might praise the graph and suggest adding labels, while a stronger response would notice that the schools have different numbers of students and prompt the student to consider whether raw counts or suspension rates better support the claim.
"AI is more than happy to just do the work for you, but we don't want to do that," Agrawal said. "We want AI to enable your thinking."
The iterative data pieces that the researchers collect — 1,200 in total, which will be stripped of any student identity — will help Agrawal establish the benchmark.
Some of the data will be used to teach an internal model what constitutes a good example of feedback, but most will be used to test whether the models have learned. Rather than producing a single score for whether an AI response is "good," the benchmark will examine questions such as: Did the system accurately interpret the student’s reasoning? Did it choose an appropriate next instructional move? Did the generated feedback actually carry out that move? Did it preserve meaningful mathematical work for the student? And did the system change its response appropriately when the student demonstrated new understanding?
In a world of synthetic, contaminated AI-generated data, the researchers said authentic samples of real student work and real teacher feedback are like gold.
"Once we've developed our model, we want to use this to test any other new model's ability," Agrawal said. "For a school or district wanting to adopt an AI model in the future, this project will provide an independent way to examine specific capabilities and limitations that matter when AI systems respond to student reasoning."
For the researchers, that distinction is fundamental. The question is not whether AI can solve the math, but whether an AI system can recognize the thinking a student has already done and support the student in doing the thinking that comes next.