Build an AI Tutor That Chooses the Next Problem

Learn how to build an AI tutor that chooses the next problem using learning signals, LLM-guided RL, and classroom-ready sequencing.

If you are building an AI tutor, the big breakthrough may not be the chatbot’s explanations. It may be the sequencing engine that decides what problem comes next. That is the central lesson from the University of Pennsylvania study on Python tutoring: students improved most when the system continuously adapted practice difficulty based on their learning signals, not when it merely responded conversationally. For EdTech teams, that shifts the product question from “How do we make the tutor sound smart?” to “How do we keep each learner in the right challenge band?”

This guide translates that research into practical engineering and curriculum design decisions. We will cover what data signals to capture, how to design a personalized sequencing loop, how to combine LLMs with reinforcement learning for problem selection, and how to deploy the system responsibly in classrooms. If you are also thinking about broader product design, compare the logic here with how teams build resilient learning products in areas like digital study systems and student engagement tools such as puzzle-based learning.

1) What the Penn study really tells EdTech builders

The key insight is sequencing, not just explanation

The Penn researchers worked with close to 800 high school students learning Python and kept the tutor’s answer behavior constrained so it would not give away solutions. The difference between groups was problem progression: one group saw a fixed easy-to-hard path, while the other received a sequence tailored in real time. The personalized path produced stronger final exam performance, suggesting that adaptive practice can outperform a one-size-fits-all curriculum even when both groups use the same LLM-based tutor. That matters because many products focus on conversational polish while leaving sequencing static.

Why this maps to the zone of proximal development

The study’s logic aligns with the zone of proximal development: the sweet spot where learners are stretched but not overwhelmed. Problems that are too easy can reduce attention and motivation, while problems that are too hard can trigger frustration and abandonment. An effective AI tutor should therefore act like a skilled classroom teacher who senses readiness, detects confusion early, and nudges the next task just enough to preserve productive struggle. That same principle is behind strong adaptive systems in other domains, from day-one retention design to high-stakes game feedback loops.

What “6 to 9 months of schooling” should and should not mean

The researchers described the gain as roughly equivalent to 6 to 9 months of extra schooling, but that estimate is not a universal promise. It was a statistical translation from one context, one subject, and one implementation. EdTech teams should treat it as evidence that sequencing changes can create meaningful lift, not as a guaranteed benchmark for every product. The right operational takeaway is simpler: if you are already collecting student interaction data, you may be able to improve outcomes significantly without rebuilding your entire content library.

2) The learning signals your AI tutor must capture

Start with observable signals, not hidden assumptions

A sequencing engine is only as good as the signals it receives. At minimum, capture response correctness, number of attempts, hint usage, edit distance between attempts, latency between steps, and time-on-task. These behaviors tell you far more than final correctness alone because they reveal struggle, confidence, persistence, and over-reliance on hints. If you want a more robust view of progress, pair those event logs with curriculum tags, skill dependencies, and item difficulty estimates.

Edits, hints, and time-on-task tell different stories

Edits are especially valuable in coding, writing, math, and other domains where partial work is informative. A learner who changes one line repeatedly may be close to mastery but missing a syntax rule, while a learner who rewrites an entire solution may be conceptually lost. Hints indicate where uncertainty appears, but they must be interpreted carefully: a hint can reflect productive curiosity or dependency behavior. Time-on-task helps distinguish thoughtful struggle from disengagement, yet it should not be used alone because some learners are fast and accurate while others are careful and slower.

Build signal quality into your instrumentation plan

Before you train anything, define a learning event schema. Each interaction should have timestamps, item IDs, skill labels, hint IDs, solution revisions, and outcome labels. Add context fields like device type, session length, grade band, and whether the learner is in class, after school, or at home. This level of instrumentation is similar to how strong analytics teams structure event data for product optimization, as seen in articles on AI-driven traffic tracking and transaction data integrity, except here your “attribution” problem is educational rather than commercial.

Pro tip: If your tutor cannot explain why it chose a problem, your curriculum team will not trust it, and neither will teachers. Log the top three reasons for each selection decision from day one.

3) Designing the data model for adaptive practice

Separate content metadata from learner state

Adaptive systems fail when item difficulty is mixed with learner proficiency in one opaque score. Instead, store content metadata separately from learner state. Content metadata should include skill tags, prerequisite graph position, cognitive demand, estimated difficulty, common misconceptions, and format type. Learner state should include mastery probabilities by skill, recency of practice, error patterns, hint dependence, pace, and confidence proxy scores. This separation makes it easier for curriculum designers to audit the system and for engineers to debug bad recommendations.

Represent skills as a graph, not a flat list

Real curriculum design is hierarchical. Students do not simply “learn Python”; they learn variables, conditionals, loops, debugging, and algorithmic thinking in a dependency sequence. A skill graph allows the tutor to recommend the next item based on prerequisite readiness, not just global difficulty. That approach is also easier to explain to teachers, who need to know whether a student is blocked because of a single missing prerequisite or because they need broader review.

Use mastery estimates, but keep them conservative

Many teams jump straight to Bayesian Knowledge Tracing or deep knowledge tracing without thinking through the failure modes. These models can be useful, but they are fragile when the dataset is small or the content is noisy. Start by estimating mastery conservatively, then update only when you see multiple consistent signals: correct performance across varied contexts, reduced hint use, shorter but accurate solution paths, and success on delayed review. For broader instructional design context, think of this as the digital equivalent of sequencing in a strong study routine, much like the logic behind low-stress digital study systems and scaffolded challenge in achievement-based learning.

4) How to implement personalized sequencing algorithms

Start with rules, then graduate to models

Do not begin with a fully autonomous RL agent. The best path for most EdTech teams is a staged system: first a rules-based baseline, then an LLM-guided selector, then a reinforcement learning policy that learns from outcomes. Rules let you encode obvious pedagogical constraints such as prerequisite order, review spacing, and safety filters. Once you have enough usage data, the model can learn which problem types generate the best next-step learning for different learner profiles.

Where the LLM fits in the architecture

An LLM should not be the sole decider of problem order. Its strengths are language understanding, item analysis, misconception classification, and generating rationales. For example, the LLM can summarize a student’s recent errors and suggest whether they likely need debugging practice, worked examples, or transfer tasks. But the actual decision engine should be constrained by curriculum rules and downstream policy logic so the tutor does not recommend advanced problems before a prerequisite is stable.

Reinforcement learning selects for learning gain, not immediate correctness

In an adaptive practice system, the reward function matters more than the algorithm label. If you optimize only for immediate accuracy, the tutor may keep serving easy items. If you optimize only for time spent, it may frustrate learners with too much difficulty. Better reward definitions include delayed post-test improvement, retention after 24 to 72 hours, reduced hint dependence, and successful transfer to new item formats. That is the core of LLM-guided reinforcement learning: the LLM helps interpret student context, while RL learns the sequence policy that maximizes durable learning, not just short-term engagement.

Use a two-stage policy for safety and control

A practical architecture is candidate generation plus policy ranking. First, generate a small set of eligible next problems using prerequisite constraints and content rules. Then rank those candidates using a learned model that estimates expected learning gain. This design keeps the system explainable and prevents the tutor from selecting nonsense items simply because a model misread a context window. It also helps curriculum teams intervene when they need to enforce a unit review or a teacher-assigned lesson path.

5) Curriculum design for AI tutors: how humans and machines should divide labor

Teachers define the learning map; the model chooses the step

The AI tutor should not invent the curriculum. Teachers, subject-matter experts, and instructional designers should define the learning map, unit boundaries, common misconceptions, mastery thresholds, and assessment cadence. The model then operates inside that map, choosing the next best step within a bounded instructional plan. This mirrors how strong product teams separate strategy from execution in other fields, as seen in guides like building a winning resume or stacking strategy under uncertainty, except here the stakes are student learning rather than personal branding or betting risk.

Design for mastery, review, and transfer

Good sequencing should not move forward forever. It should cycle through introduction, practice, spaced review, and transfer tasks. If a student masters a concept in a narrow context, the system should later reintroduce it in a different format to test retention. That means the tutor needs a memory model of what was learned, how recently it was practiced, and whether the learner has demonstrated the skill under more complex conditions. Without this, the tutor may confuse short-term performance with durable understanding.

Plan for mixed classrooms, not idealized individuals

In real classrooms, a single tutor may serve learners at very different starting points. Your content design must therefore support branching instruction, enrichment, and remediation without making the teacher manage separate systems manually. Build lesson-level controls so teachers can pin certain skills, lock sequence ranges, or assign a diagnostic checkpoint before the adaptive engine resumes. That keeps personalized sequencing aligned with classroom pacing and reduces the fear that AI is “taking over” instruction.

6) A practical implementation stack for EdTech teams

Recommended system layers

A production AI tutor typically needs five layers: content repository, learner event store, feature extraction service, sequencing policy, and tutoring interface. The content repository holds tagged items and explanations. The event store captures every action, edit, hint request, and timing signal. Feature extraction transforms raw events into mastery indicators, while the policy layer decides what comes next. The interface then presents the item and collects the next round of data.

Choose observability over sophistication early on

It is tempting to over-engineer the model layer before the product has stable telemetry. Resist that temptation. Use dashboards that show item difficulty curves, hint rates, dropout points, and learning gain by sequence type. Compare fixed versus adaptive cohorts, and segment by grade, device, and prior achievement. These operational habits are similar to disciplined measurement practices in other technology domains, such as uncertainty estimation and data storage architecture, where visibility is the difference between confidence and guesswork.

Because your tutor will process detailed behavioral data, privacy engineering is not optional. Minimize identifiable data, encrypt event streams, define retention windows, and document how learner signals are used for personalization. Schools will ask whether the system stores transcripts, how long it keeps logs, whether training data is separated from classroom records, and whether teachers can delete or export student data. Build for those questions now rather than retrofitting compliance later.

Implementation Layer	Purpose	Key Inputs	Typical Failure Mode	What Good Looks Like
Content repository	Stores tagged problems and explanations	Skill tags, prerequisites, difficulty, misconceptions	Poor tagging creates bad recommendations	Curriculum-reviewed metadata with version control
Learner event store	Captures behavior signals	Edits, hints, time-on-task, attempts, timestamps	Missing events or inconsistent logs	Reliable event schema and audit trail
Feature extraction	Builds learner state	Mastery estimates, error patterns, pace	Overfitting to noisy short-term behavior	Conservative, interpretable skill features
Sequencing policy	Selects next problem	Learner state, content graph, reward signals	Chooses items that are too easy or too hard	Balances challenge, review, and progress
Teacher controls	Aligns AI with classroom needs	Pinned skills, locks, pacing rules	Teachers lose trust or override too often	Transparent, easy-to-use classroom governance

7) Evaluation: how to prove the tutor is actually helping

Measure learning, not just engagement

A common mistake in edtech is to call a product effective because students used it. Usage is not evidence of learning. Your primary metrics should include post-test gain, delayed retention, transfer performance, and reduction in misconception recurrence. Engagement metrics still matter, but they should act as guardrails rather than the headline outcome. If users spend more time but learn less, the system is optimizing the wrong objective.

Use randomized and quasi-experimental designs

The Penn study is strong because students were randomly assigned to fixed versus adaptive sequencing. You should aim for similar rigor whenever possible. If randomization is not feasible, use matched comparison groups, pre/post assessments, and difference-in-differences designs. Also examine whether the system helps some learners more than others, especially students who struggle early, because those are often the learners most sensitive to personalized sequencing.

Diagnose failure by segment, not just average

Averages can hide important problems. Your tutor may help advanced learners but not beginners, or it may work well in one unit and poorly in another. Break analysis down by skill, grade, prior performance, session length, and classroom context. If possible, look at the sequence trajectories of successful learners versus those who stalled. That often reveals whether the engine is too aggressive, too conservative, or too dependent on a narrow set of signals.

Pro tip: When adaptive sequencing underperforms, the bug is often not the model. It is usually a curriculum mismatch, a weak skill graph, or an item bank that does not have enough medium-difficulty problems.

8) Classroom deployment best practices

Introduce the system as a coach, not a shortcut

Teachers and students need a clear mental model. Tell students the AI tutor is there to choose the next practice step, not to replace thinking. In classrooms, position the tool as a guided practice partner that helps them stay in the productive middle of difficulty. This framing is essential because students can otherwise over-trust the system or use it as a way to bypass struggle, the exact risk noted in many skeptical studies of chatbot tutoring.

Train teachers on intervention points

Teachers should know when to trust the sequence and when to intervene. For example, if a student repeatedly requests hints, stalls for long periods, or cycles through similar errors, the teacher may need to provide a mini-lesson or group conference. Build teacher dashboards that surface these triggers without overwhelming staff with raw logs. The goal is not to make educators analyze every click; it is to help them act on the few signals that matter most.

Start with one subject and one unit

Do not roll out adaptive sequencing across every subject at once. Begin with a domain that has clear prerequisite structure and measurable problem-solving behavior, such as coding, algebra, or science practice. Pilot one unit, collect enough interaction data to tune the policy, and expand only after you can show stable learning gains. The Penn Python study is useful precisely because programming practice naturally produces rich learning signals, from edits and retries to hint patterns and completion time.

9) Common pitfalls and how to avoid them

Optimizing for the wrong reward

If you reward the model for short-term accuracy, it may become overly conservative. If you reward completion speed, it may skip necessary review. If you reward hint reduction alone, it may make students feel stuck. A balanced reward function should value learning gain, stable retention, and productive challenge. This is one reason why a human-in-the-loop curriculum team is still necessary even when the model is sophisticated.

Confusing conversational personalization with pedagogical personalization

A chatbot can feel personal simply because it responds to the student’s wording. But true personalization requires knowing what the student should do next, not just replying in a tailored tone. That distinction is the heart of the Penn result and a critical lesson for product teams. The student may think, “The AI understands me,” while the system is still serving the wrong next step. This is where sequencing beats surface-level conversational polish.

Underinvesting in item quality

An adaptive engine cannot fix a weak question bank. If items are ambiguous, too narrow, or poorly aligned to skills, the sequencing algorithm will learn bad patterns. Invest in item writing, pilot review, and misconception tagging just as seriously as you invest in model training. In practice, strong AI tutors are content systems first and machine-learning systems second.

10) A build roadmap for the next 90 days

Days 1–30: define the learning model

Audit your curriculum map, identify prerequisite relationships, and define the signals you can reliably capture. Decide what counts as mastery, partial progress, and struggle. Write a measurement plan that includes baseline assessments and outcome metrics. If you need a product planning reference for how to structure an implementation roadmap, useful analogies can be found in operational guides like AI partnership strategy and technology-enabled content delivery.

Days 31–60: build the baseline sequencer

Create a rules-based sequencer that respects prerequisites and difficulty bands. Add logging for every learner interaction, then run internal tests with a small content set. In parallel, build the teacher dashboard so educators can inspect sequence decisions. This stage should make the AI explainable before it becomes adaptive.

Days 61–90: launch controlled experiments

Run an A/B test comparing fixed sequencing to your adaptive baseline. Use a subset of classrooms, collect feedback from teachers, and inspect whether the tutor is over-served by easy items or stuck in loops. Once you trust the telemetry, introduce a model-based ranker and begin tuning reward functions around delayed learning outcomes. At this stage, the product starts behaving like a true adaptive practice system instead of a static exercise library.

FAQ

What is the most important learning signal for an AI tutor?

There is no single best signal. In practice, the most useful combination is correctness, time-on-task, hint usage, and edit behavior. Together, they reveal whether a student is ready to advance, needs scaffolding, or is struggling in a way that requires teacher intervention. The key is to interpret signals in context rather than treating any one metric as truth.

Should we use an LLM to choose the next problem directly?

Usually no. An LLM is best used for interpretation, explanation, and candidate generation, while a constrained policy or RL layer makes the final sequencing decision. This reduces the risk of pedagogical mistakes and makes the system easier for teachers to trust. A hybrid architecture is safer and more scalable than letting the model decide everything.

How do we keep the tutor inside the zone of proximal development?

Use mastery estimates, prerequisite graphs, and difficulty bands to avoid extremes. The tutor should present tasks that are hard enough to require effort but not so hard that they trigger repeated failure. If the learner is breezing through items, raise the challenge. If the learner is stuck, move to simpler prerequisites or targeted hints before advancing.

What kind of classroom pilot should we run first?

Start with one subject, one grade band, and one unit with clear learning objectives. Coding, algebra, and science problem sets are often good candidates because they produce rich behavior data. Keep the pilot small enough to support close teacher feedback and quick iteration. Your goal is not scale at first; it is trustworthy evidence.

How do we know the sequencing engine is actually helping?

Look for improvements on post-tests, delayed retention, and transfer tasks, not just engagement time. Compare adaptive versus fixed sequences in a controlled test, and segment results by learner type. If the adaptive group performs better across multiple outcome measures, your sequencing logic is likely adding real educational value.

Can adaptive sequencing work outside coding?

Yes. The same core design applies to math, language learning, science practice, test prep, and many skill-based domains. The content structure changes, but the underlying principles remain the same: capture learning signals, estimate readiness, choose the next best task, and evaluate based on durable progress.

Conclusion: the future of AI tutoring is sequenced, not just conversational

The Penn study is a reminder that the most meaningful AI tutoring gains may come from better pedagogy, not flashier dialogue. If your system knows how to identify the right next problem, it can keep learners in the zone where challenge and success reinforce each other. That requires disciplined data capture, a curriculum-aware content graph, cautious use of LLMs, and an RL layer optimized for long-term learning rather than instant correctness. In other words, the next generation of AI tutors will win by acting less like chatbots and more like excellent teachers with extraordinary memory.

If you are building now, focus on the pieces that make sequencing trustworthy: clean event logs, clear skill maps, teacher controls, and evaluation designs that prove impact. Then scale carefully. For further perspective on learner experience, data handling, and product governance, see also our guides on data storage and architecture, leadership and operational change, and audience feedback loops.

The quest to build a better AI tutor - The Penn study that inspired this practical sequencing guide.
How to Build a Low-Stress Digital Study System Before Your Phone Runs Out of Space - Useful for thinking about learner workflow and study habits.
How AI Forecasting Improves Uncertainty Estimates in Physics Labs - A helpful model for thinking about confidence and prediction quality.
How to Track AI-Driven Traffic Surges Without Losing Attribution - Strong analogy for logging, attribution, and data quality.
Streamlining Your Smart Home: Where to Store Your Data - A practical reference for designing storage and retention policies.

Jordan Mercer

Senior EdTech Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.