# Track 03 — Workflows Research: AI Adaptive Learning

## Source Manifest

| source_id | url | bucket | last_checked | author/host | one-line note |
|-----------|-----|--------|--------------|-------------|---------------|
| T03-S001 | https://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf | verified_primary | 2026-05-25 | CMU PACT Center | KLI Framework paper — KC analysis methodology |
| T03-S002 | https://pslcdatashop.web.cmu.edu/about/casestudies.html | verified_primary | 2026-05-25 | CMU DataShop | Learning curve analysis case studies for KC validation |
| T03-S003 | https://www.numberanalytics.com/blog/10-steps-master-item-response-theory-assessments | surrogate_primary | 2026-05-25 | Number Analytics | vendor docs (EdTech 平台) — 10-step IRT mastery guide |
| T03-S004 | https://pmc.ncbi.nlm.nih.gov/articles/PMC11561393/ | verified_primary | 2026-05-25 | NIH PMC | irtQ R package for IRT calibration |
| T03-S005 | https://arxiv.org/html/2409.08823v1 | verified_primary | 2026-05-25 | arXiv | AutoIRT: automated ML-based item calibration |
| T03-S006 | https://arxiv.org/html/2105.15106v4 | verified_primary | 2026-05-25 | arXiv | Survey of Knowledge Tracing models and applications |
| T03-S007 | https://www.mdpi.com/2624-8611/5/3/50 | surrogate_primary | 2026-05-25 | MDPI (Education Sciences) | vendor docs (学术出版商) — pyBKT introduction and tutorial |
| T03-S008 | https://github.com/umass-ml4ed/dialogue-kt | verified_primary | 2026-05-25 | UMass ML4ED GitHub | Dialogue-KT: LLM-based knowledge tracing LAK2025 |
| T03-S009 | https://link.springer.com/chapter/10.1007/978-3-031-92970-0_5 | surrogate_primary | 2026-05-25 | Springer | vendor docs (学术出版商) — Curriculum Modeling for Adaptive Learning |
| T03-S010 | https://www.adaptemy.com/content-authoring-in-an-adaptive-learning-world-how-publishers-make-the-leap/ | surrogate_primary | 2026-05-25 | Adaptemy | vendor docs (EdTech 平台) — Content authoring for adaptive learning |
| T03-S011 | https://research.duolingo.com/papers/settles.acl16.pdf | surrogate_primary | 2026-05-25 | Duolingo Research | vendor docs (EdTech 平台) — Half-Life Regression paper |
| T03-S012 | https://github.com/open-spaced-repetition/fsrs4anki | verified_primary | 2026-05-25 | open-spaced-repetition GitHub | FSRS algorithm for Anki |
| T03-S013 | https://github.com/open-spaced-repetition/srs-benchmark | verified_primary | 2026-05-25 | open-spaced-repetition GitHub | SRS benchmark suite |
| T03-S014 | https://pmc.ncbi.nlm.nih.gov/articles/PMC6410796/ | verified_primary | 2026-05-25 | NIH PMC | Enhancing human learning via spaced repetition optimization |
| T03-S015 | https://xapi.com/learning-record-store/ | surrogate_primary | 2026-05-25 | xAPI.com (Rustici) | vendor docs (EdTech 平台) — LRS architecture guide |
| T03-S016 | https://www.academyofmine.com/advanced-learning-analytics-with-xapi-and-lrs/ | surrogate_primary | 2026-05-25 | Academy of Mine | vendor docs (EdTech 平台) — xAPI analytics tutorial |
| T03-S017 | https://pmc.ncbi.nlm.nih.gov/articles/PMC8853217/ | verified_primary | 2026-05-25 | NIH PMC | Learning analytics dashboard: actionable insights |
| T03-S018 | https://www.assistments.org/research | surrogate_primary | 2026-05-25 | ASSISTments Foundation | vendor docs (EdTech 平台) — research and RCT evidence |
| T03-S019 | https://educationaldatamining.org/edm2024/proceedings/2024.EDM-tutorials.132/index.html | surrogate_primary | 2026-05-25 | EDM Society | vendor docs (协会) — EDM 2024 tutorial on A/B test planning |
| T03-S020 | https://arxiv.org/pdf/2306.06273 | verified_primary | 2026-05-25 | arXiv | Auxiliary data for A/B test precision on educational platforms |
| T03-S021 | https://standards.1edtech.org/ | surrogate_primary | 2026-05-25 | 1EdTech Consortium | vendor docs (协会) — LTI/xAPI/QTI standards portal |
| T03-S022 | https://www.imsglobal.org/spec/lti/v1p3/impl | surrogate_primary | 2026-05-25 | 1EdTech / IMS Global | vendor docs (协会) — LTI Advantage 1.3 implementation guide |
| T03-S023 | https://www.imsglobal.org/spec/lti/v1p3/cert | surrogate_primary | 2026-05-25 | 1EdTech / IMS Global | vendor docs (协会) — LTI 1.3 conformance certification guide |
| T03-S024 | https://www.hireplicity.com/blog/ferpa-compliance-checklist-2025 | surrogate_primary | 2026-05-25 | Hireplicity | vendor docs (EdTech 平台) — FERPA compliance checklist 2025 |
| T03-S025 | https://6b.education/insight/building-privacy-compliant-systems-edtech-development-under-gdpr-coppa-and-ferpa/ | surrogate_primary | 2026-05-25 | 6B Education | vendor docs (EdTech 平台) — Privacy-compliant EdTech architecture |
| T03-S026 | https://studentprivacy.ed.gov/ | verified_primary | 2026-05-25 | US Dept of Education | FERPA/student privacy official portal |
| T03-S027 | https://pmc.ncbi.nlm.nih.gov/articles/PMC12453719/ | verified_primary | 2026-05-25 | NIH PMC | LPITutor: LLM personalized tutoring with RAG |
| T03-S028 | https://blog.khanacademy.org/khan-academys-framework-for-responsible-ai-in-education/ | surrogate_primary | 2026-05-25 | Khan Academy Blog | vendor docs (EdTech 平台) — Responsible AI framework |
| T03-S029 | https://www.datadoghq.com/blog/llm-guardrails-best-practices/ | surrogate_primary | 2026-05-25 | Datadog | vendor docs (EdTech 平台) — LLM guardrails best practices |
| T03-S030 | https://www.eleapsoftware.com/xapi-in-lms-systems-complete-integration-guide/ | surrogate_primary | 2026-05-25 | eLeaP Software | vendor docs (EdTech 平台) — xAPI in LMS integration guide |
| T03-S031 | https://mrccedtech.com/edtech-compliance-checklist-ferpa-coppa-and-student-data-privacy/ | surrogate_primary | 2026-05-25 | MRCC EdTech | vendor docs (EdTech 平台) — EdTech compliance checklist |
| T03-S032 | https://devoxsoftware.com/blog/the-next-wave-of-adaptive-learning-and-strategic-roadmap-2026/ | surrogate_primary | 2026-05-25 | Devox Software | vendor docs (EdTech 平台) — Adaptive learning roadmap 2026 |
| T03-S033 | https://html.rhhz.net/tis/html/202105036.htm | surrogate_primary | 2026-05-25 | 清华大学学报 (智能系统) | vendor docs (学术出版商) — 面向智能教育的自适应学习关键技术与应用 |
| T03-S034 | https://arxiv.org/html/2601.01708v1 | verified_primary | 2026-05-25 | arXiv | Thinking-KT: training-free LLM-based knowledge tracing |

---

### 1. Knowledge Component Analysis and Curriculum Mapping

- **Trigger**: New course or module needs adaptive treatment
- **Input**: Raw curriculum standards, learning objectives, existing content
- **Output**: KC model (knowledge component map) + prerequisite graph

**入门 SOP**: 1. Gather curriculum standards and learning objectives from subject-matter experts (SMEs). 2. Decompose each objective into fine-grained knowledge components (KCs) — typically 5-15 KCs per lesson unit, following the KLI Framework principle that each KC should be independently learnable and assessable. 3. Define prerequisite relationships between KCs using expert judgment to create a directed acyclic graph (DAG). 4. Map each KC to existing content items and assessment items (KC-to-content mapping table). 5. Pilot with a small student cohort (n >= 30) and collect interaction data. 6. Run learning curve analysis on pilot data using CMU DataShop or equivalent — smooth power-law decay confirms correct KC granularity; flat or erratic curves indicate KCs need splitting or merging. 7. Iterate KC model with domain experts based on data. 8. Freeze KC model version and document in curriculum spec.

- **资深路径**: 跳过 manual decomposition for well-studied domains — import existing KC models from PSLC DataShop or published Q-matrices. 优化 prerequisite graph using automated Q-matrix refinement (e.g., SPARFA, NMF-based methods) on historical interaction data. 额外 consider cross-domain transfer KCs (e.g., "proportional reasoning" shared across math and science), and use LLM-assisted KC proposal generation from textbook content as a starting draft for expert review.
- **AI 变化 (2024-2025)**: GPT-4/Claude can generate initial KC decompositions from syllabi or textbook chapters in minutes (vs. days of expert work), but expert validation remains mandatory — LLMs consistently produce KCs at too coarse a grain without domain-specific tuning. LLM-generated prerequisite suggestions from course descriptions show ~70% agreement with expert graphs but miss non-obvious dependencies. Automated Q-matrix refinement now handles datasets of 100K+ interactions efficiently.
- **last_updated**: 2025-06
- **Decay risk**: medium — KC models are stable for established subjects (algebra, physics), but AI-assisted authoring tools are changing the speed and granularity of the process.
- **失败模式**: KCs too coarse → adaptation is ineffective (system can't distinguish what the student doesn't know); KCs too fine → severe cold-start problem (insufficient data per KC); prerequisite graph has cycles or wrong edges → students get stuck in loops or skip prerequisites; KC-to-content mapping incomplete → some KCs have no practice items.

evidence: [T03-S001, T03-S002, T03-S009]

### 2. Item Authoring and Psychometric Calibration

- **Trigger**: Item bank needs new items or existing items need parameter estimation
- **Input**: Learning objectives, KC map, draft items from SMEs
- **Output**: Calibrated item bank with IRT parameters (difficulty, discrimination, guessing)

**入门 SOP**: 1. Write item specifications aligned to KC map — each item targets 1-3 KCs, with clear stems, distractors (for MC), and rubrics. 2. Conduct content review with SMEs for accuracy, bias, and alignment to standards. 3. Pilot-test items with a calibration sample (n >= 200 for 2PL, n >= 500 for 3PL) using a balanced incomplete block design or equivalent. 4. Run IRT calibration using mirt (R) or irtQ — estimate item parameters (difficulty b, discrimination a, guessing c). 5. Evaluate item fit: check infit/outfit MNSQ (acceptable range 0.7-1.3), examine item characteristic curves (ICCs) for anomalies. 6. Flag and remove items showing differential item functioning (DIF) across demographic subgroups. 7. Check local independence assumptions — items sharing a stimulus may violate this. 8. Bank surviving items with metadata (KC tags, IRT params, exposure controls, content tags) into the item bank system (e.g., TAO, custom DB). 9. Set exposure control parameters to prevent item overuse in CAT deployments.

- **资深路径**: 跳过 full pilot for item variants that differ only in surface features (isomorphic items) — use anchored calibration from parent item parameters. 优化 calibration with AutoIRT (ML-based automated calibration) for large item banks (1000+ items), reducing manual parameter review. 额外 implement continuous calibration (online IRT) where item parameters update as new response data accumulates, rather than batch recalibration cycles.
- **AI 变化 (2024-2025)**: LLMs (GPT-4, Claude) can generate item stems and distractors from KC specifications, accelerating authoring 5-10x; however, generated items still require psychometric calibration — LLM-estimated difficulty correlates only r=0.4-0.6 with empirical difficulty. AutoIRT (arXiv 2024) uses ML to automate calibration decisions previously requiring psychometricians. LLM-reconstructed item characteristic curves (arXiv 2025) show promise for zero-shot difficulty estimation but are not production-ready.
- **last_updated**: 2025-06
- **Decay risk**: medium — IRT fundamentals are stable (50+ year methodology), but AI-assisted authoring and automated calibration are rapidly changing the authoring end of the pipeline.
- **失败模式**: Calibration sample too small → unstable parameter estimates, especially for 3PL guessing parameter; DIF not checked → biased items disadvantage subgroups; exposure controls absent → item compromise in high-stakes CAT; items not tagged to KCs → adaptive engine cannot select appropriately; local dependence ignored → inflated reliability.

evidence: [T03-S003, T03-S004, T03-S005]

### 3. Student Model Training and Validation

- **Trigger**: Sufficient interaction data accumulated or new adaptive module launch
- **Input**: Student interaction logs (attempts, correctness, timestamps, KC tags)
- **Output**: Trained knowledge tracing model + mastery probability estimates per student-KC

**入门 SOP**: 1. Extract and clean interaction logs — each row contains (student_id, KC_id, timestamp, correct/incorrect, response_time). Remove test/admin accounts, deduplicate, handle missing KC tags. 2. Split data: 80% train / 20% held-out test, splitting by student (not by interaction) to prevent leakage. 3. Choose baseline model: BKT (pyBKT) for interpretable per-KC mastery; DKT (LSTM/Transformer) for higher predictive accuracy when data is abundant (>50K interactions). 4. Train model — for BKT: fit four parameters per KC (p_init, p_learn, p_guess, p_slip) using EM; for DKT: train RNN on interaction sequences with binary cross-entropy loss. 5. Evaluate on held-out set using AUC-ROC (target >= 0.75) and RMSE. Compare against majority-class baseline. 6. Conduct ablation: check per-KC AUC — KCs with AUC < 0.60 may indicate bad KC definitions (loop back to Workflow 1). 7. Validate mastery thresholds: confirm that students flagged as "mastered" (P(mastery) >= 0.95 for BKT) actually perform well on subsequent assessments. 8. Package model for deployment: serialize parameters, document version, set up inference API endpoint.

- **资深路径**: 跳过 BKT for domains with complex skill interactions — use attention-based models (AKT, SAINT, simpleKT) that capture cross-KC dependencies. 优化 hyperparameters with Bayesian optimization rather than grid search; use DKVMN for explicit key-value memory of concept states. 额外 implement ensemble of BKT + DKT, using BKT for interpretability/cold-start and DKT for prediction accuracy; evaluate with causal learning curves (not just predictive metrics).
- **AI 变化 (2024-2025)**: Thinking-KT (arXiv 2025) demonstrates training-free knowledge tracing using LLM reasoning — no model training needed, just prompt the LLM with student history. Dialogue-KT (LAK 2025) extends KT to free-form tutoring dialogues, not just discrete right/wrong. LLM-based KT achieves competitive AUC on standard benchmarks but at 100x inference cost. Hybrid approaches (LLM for cold-start, trained model for steady-state) are emerging as practical.
- **last_updated**: 2025-06
- **Decay risk**: high — model architectures evolving rapidly (new variants every 3-6 months); LLM-based approaches may displace trained models for low-data regimes within 12-18 months.
- **失败模式**: Data leakage (splitting by interaction not student) → inflated AUC; mastery threshold too low → students advance prematurely; mastery threshold too high → student frustration and over-practice; KC tags wrong in training data → model learns noise; model not retrained as curriculum changes → drift.

evidence: [T03-S006, T03-S007, T03-S008, T03-S034]

### 4. Adaptive Content Delivery Engine Configuration

- **Trigger**: Student model and item bank ready for real-time deployment
- **Input**: KC prerequisite graph, calibrated item bank, trained student model
- **Output**: Real-time content selection engine serving personalized learning paths

**入门 SOP**: 1. Define the outer loop (problem/topic selection) and inner loop (step-level hints/feedback) architecture per VanLehn's framework. 2. Implement mastery-based sequencing: for each student, query the student model for P(mastery) per KC → select next KC where prerequisites are mastered but target KC is below threshold. 3. Within the selected KC, choose an item from the calibrated bank matching the student's estimated ability (using CAT item selection: maximum information, or Bayesian optimal). 4. Implement the inner loop: for multi-step items, provide hint sequences and step-level feedback; for single-response items, provide immediate correctness feedback + worked-example fallback after N consecutive errors. 5. After each student response, update the student model in real-time (online BKT update or DKT forward pass). 6. Implement stopping rules: advance to next KC when P(mastery) >= threshold; escalate to human instructor when student is stuck (>= 5 consecutive errors on same KC). 7. Configure content diversity constraints: avoid showing the same item twice, balance item types (MC, short-answer, simulation), respect exposure controls. 8. Load-test the delivery engine for target concurrency (e.g., 1000 simultaneous students, <200ms response time per request).

- **资深路径**: 跳过 simple mastery-based sequencing — implement knowledge-space-based delivery (ALEKS style) where the system identifies the student's entire knowledge state and offers items from the "outer fringe" of feasible learning. 优化 with multi-armed bandit or reinforcement learning for item selection (explore/exploit tradeoff between learning gain and model uncertainty). 额外 implement "zone of proximal development" targeting — select items at ~70-85% predicted success probability for optimal challenge; integrate multi-modal content (video, simulation, text) selection based on learner preference modeling.
- **AI 变化 (2024-2025)**: Knowledge graphs for content organization are becoming standard, with LLMs used to auto-tag content with KC metadata and difficulty estimates. Generative AI enables on-the-fly content creation — instead of selecting from a fixed item bank, LLMs generate personalized practice problems. Reinforcement learning from human feedback (RLHF) is being explored for optimizing content sequencing policies. Real-time content generation introduces new latency/quality tradeoffs.
- **last_updated**: 2025-06
- **Decay risk**: high — the shift from static item banks to generative content delivery is accelerating; RL-based sequencing still experimental but gaining traction.
- **失败模式**: Mastery threshold miscalibrated → premature advancement or over-drilling; inner loop too aggressive with hints → students learn to game hints instead of thinking; no diversity constraints → student boredom from repetitive content; latency too high → student disengagement; outer loop ignores prerequisites → students face items they lack foundations for.

evidence: [T03-S009, T03-S010, T03-S032, T03-S033]

### 5. Spaced Repetition Schedule Optimization

- **Trigger**: Retention-critical content (vocabulary, facts, formulas) identified
- **Input**: Review history (timestamps, outcomes), memory model parameters
- **Output**: Optimized review schedule per learner-item pair

**入门 SOP**: 1. Identify retention-critical content: facts, vocabulary, formulas, definitions — items where forgetting is the primary barrier, not conceptual misunderstanding. 2. Choose a memory model: Leitner (simple box system), SM-2 (Anki default), Half-Life Regression (HLR, Duolingo), or FSRS (state-of-the-art open-source). 3. Initialize parameters: for HLR, set default half-life based on item difficulty and learner history; for FSRS, use default parameters from the pre-trained model. 4. Deploy review scheduling: when a learner logs in, compute recall probability for each due item using the memory model → sort by urgency (lowest recall probability first) → present top N items for review. 5. After each review, update the memory model: correct recall → increase stability (longer next interval); incorrect → decrease stability (shorter interval). 6. Monitor aggregate retention rate: target 85-90% recall at review time (the "desirable difficulty" zone). 7. A/B test algorithm variants: compare new scheduler against baseline on retention rate at 7-day and 30-day follow-up. 8. Tune hyperparameters based on population-level data; optionally personalize per learner.

- **资深路径**: 跳过 SM-2 or Leitner entirely — deploy FSRS v7 (2025) which has been benchmarked on 500M+ Anki reviews and reduces review load by 20-30% vs SM-2 at equivalent retention. 优化 with per-learner parameter fitting using the learner's own review history (FSRS optimizer). 额外 implement "desired retention" as a user-configurable parameter (e.g., medical students choose 95%, casual learners choose 85%); integrate spaced repetition into the adaptive engine's outer loop so review items are interleaved with new learning.
- **AI 变化 (2024-2025)**: FSRS v5-v7 evolution represents the most active open-source development in SRS algorithms, with each version improving prediction accuracy. LLMs are being explored for generating spaced repetition flashcards automatically from course content. The open-spaced-repetition project's benchmark suite allows objective comparison of any scheduling algorithm. Duolingo's Birdbrain system (HLR successor) uses deep learning for retention prediction but remains proprietary.
- **last_updated**: 2025-06
- **Decay risk**: medium — core memory science (exponential forgetting, spacing effect) is stable; algorithm implementations are evolving rapidly (FSRS versions every 6 months).
- **失败模式**: Intervals too short → review overload, learner burnout; intervals too long → forgetting before review, low retention; no personalization → one-size-fits-all schedule suboptimal for fast/slow learners; ignoring item difficulty → hard items under-reviewed; not distinguishing recall failure types (tip-of-tongue vs. total blank) → suboptimal model updates.

evidence: [T03-S011, T03-S012, T03-S013, T03-S014]

### 6. Learning Analytics Dashboard and Reporting

- **Trigger**: Adaptive system generating xAPI/event data; teachers/admins need insights
- **Input**: xAPI statements from LRS, student model outputs, assessment results
- **Output**: Actionable dashboards for teachers, admins, and learners

**入门 SOP**: 1. Define xAPI statement vocabulary: map every trackable learning event to actor-verb-object-result triples (e.g., "student X attempted problem Y with result Z"). Use ADL xAPI vocabulary or define custom verbs for adaptive-specific events (mastered, reviewed, hint-requested). 2. Instrument the adaptive platform to emit xAPI statements to a Learning Record Store (LRS) — options include Watershed (commercial), Learning Locker (open-source), or Ralph (lightweight). 3. Design dashboard views by stakeholder: teachers need class-level mastery heatmaps and at-risk student alerts; admins need cohort completion rates and efficacy metrics; learners need personal progress and upcoming reviews. 4. Build ETL pipeline: LRS → data warehouse (scheduled pull or streaming) → BI layer (Metabase, Tableau, or custom React dashboard). 5. Implement alert rules: flag students with >3 days inactivity, students with KC mastery regression, or students stuck on a single KC for >N attempts. 6. Co-design dashboards with teachers via iterative prototyping — research shows teacher-co-designed dashboards have significantly higher adoption. 7. Validate that dashboard insights are actionable: each metric should suggest a concrete intervention (e.g., "5 students need review on fractions" → teacher assigns targeted practice). 8. Monitor dashboard usage analytics to confirm adoption.

- **资深路径**: 跳过 basic descriptive analytics — implement predictive analytics (dropout risk prediction, grade forecasting) using student model outputs + engagement features. 优化 with LLM-generated narrative summaries of student performance (instead of just charts) for teachers who prefer text. 额外 implement prescriptive analytics: the dashboard not only shows "student X is struggling with KC Y" but recommends a specific intervention with evidence for its effectiveness.
- **AI 变化 (2024-2025)**: LLM-driven dashboard narratives (arXiv 2024) auto-generate natural language summaries from xAPI data, replacing manual report writing. AI-powered anomaly detection identifies unusual learning patterns without manual threshold setting. Integration of multimodal data (video engagement, eye-tracking from webcam) into analytics pipelines is emerging in research but not yet standard in production.
- **last_updated**: 2025-06
- **Decay risk**: medium — xAPI standard is stable (1.0.3 current); dashboard tooling evolving but patterns well-established; LLM-generated insights are the main area of flux.
- **失败模式**: Dashboard shows data but not actionable insights → teacher ignores it; alert fatigue from too many false positives → critical alerts missed; xAPI statements inconsistently generated → incomplete data; dashboard latency too high for real-time use → teachers check it rarely; privacy violation if student-level data exposed to wrong stakeholders.

evidence: [T03-S015, T03-S016, T03-S017, T03-S030]

### 7. A/B Testing and Efficacy Evaluation

- **Trigger**: New pedagogical intervention, algorithm change, or content variant to test
- **Input**: Hypothesis, target population, outcome measures, existing platform traffic
- **Output**: Causal effect estimate with confidence intervals, go/no-go decision

**入门 SOP**: 1. Define hypothesis precisely: "Intervention X will improve outcome Y by Z% compared to control" — outcomes should be learning gains (pre-post test), retention rates, or completion rates, not just engagement metrics. 2. Power analysis: calculate required sample size using expected effect size (Cohen's d = 0.2 for small educational effects is common), desired power (0.80), and alpha (0.05). For within-platform A/B tests, use auxiliary data to reduce required sample by 20-50%. 3. Design randomization: random assignment at student level (simple) or classroom level (cluster RCT — required when intervention is classroom-wide). For cluster RCTs, account for intraclass correlation (ICC ~ 0.1-0.3 typical in education). 4. Implement treatment assignment in platform: use feature flags or experiment framework to route students to treatment/control. 5. Run experiment for planned duration — resist peeking at results (multiple comparisons problem). Use sequential testing or Bayesian methods if early stopping is needed. 6. Analyze: ITT (intention-to-treat) as primary analysis; CACE (complier average causal effect) as secondary. Report effect size (Cohen's d or Hedges' g), 95% CI, and p-value. 7. Check for differential attrition between arms — >5% differential attrition threatens internal validity. 8. Document results and make go/no-go decision; for successful interventions, roll out to full population with monitoring.

- **资深路径**: 跳过 simple two-arm designs — implement multi-arm bandits (Thompson sampling) for continuous optimization across many variants simultaneously, or factorial designs to test multiple factors. 优化 with CUPED (controlled-experiment using pre-experiment data) or related variance reduction techniques to get same statistical power with 20-50% fewer students. 额外 implement the ASSISTments model of "micro-RCTs" embedded in daily homework — every homework assignment is a potential experiment, enabling rapid iteration.
- **AI 变化 (2024-2025)**: Bayesian approaches to effect size estimation (arXiv 2025) provide better uncertainty quantification for small samples common in education. AI-powered experiment design tools auto-suggest sample sizes and detect threats to validity. LLM-based literature review can rapidly identify prior effect sizes for power analysis. Automated heterogeneous treatment effect analysis identifies which student subgroups benefit most.
- **last_updated**: 2025-06
- **Decay risk**: low — RCT methodology is foundational and stable; tools and variance reduction techniques are incremental improvements.
- **失败模式**: Underpowered study → null result interpreted as "no effect" (Type II error); peeking at results → inflated false positive rate; randomization broken (students self-select) → confounded estimates; measuring engagement instead of learning → optimizing for the wrong thing; not accounting for clustering → underestimated standard errors and false positives; Hawthorne effect in short-duration studies.

evidence: [T03-S018, T03-S019, T03-S020]

### 8. LMS and Platform Integration (LTI/xAPI)

- **Trigger**: Adaptive module needs to work inside institutional LMS
- **Input**: Adaptive tool codebase, target LMS(es), institutional IT requirements
- **Output**: LTI 1.3 certified integration with xAPI event telemetry

**入门 SOP**: 1. Register as an LTI 1.3 Tool Provider: generate RSA key pair, configure OIDC login initiation URL, and tool launch URL. 2. Implement the LTI 1.3 launch flow: OIDC third-party login → platform returns id_token (JWT) with user identity, course context, and roles. 3. Implement LTI Advantage services as needed: Deep Linking (for content selection), Assignment and Grade Services (AGS — for grade passback), Names and Role Provisioning Services (NRPS — for roster sync). 4. Map LTI user identifiers to internal student IDs in the adaptive system — handle the case where a student accesses from multiple LMS instances. 5. Implement xAPI statement emission: on each learning event, send xAPI statement to the institutional LRS or your own LRS, including the LTI context (course_id, section) as xAPI context extensions. 6. Test integration against the 1EdTech LTI Reference Implementation — run conformance test suite. 7. Fix conformance failures, then submit for 1EdTech certification (optional but strongly recommended for institutional sales). 8. Document integration guide for LMS admins: key exchange process, launch URL configuration, grade passback setup, and troubleshooting.

- **资深路径**: 跳过 manual key exchange — implement dynamic registration (LTI 1.3 Dynamic Registration spec) so LMS admins can add the tool with one click. 优化 with pre-built LTI libraries (ltijs for Node.js, pylti1p3 for Python) instead of implementing from scratch. 额外 implement Caliper Analytics Profile alongside xAPI for institutions that use Caliper; support QTI 3.0 import/export for assessment interoperability; handle multi-tenant deployment where one tool instance serves hundreds of institutions.
- **AI 变化 (2024-2025)**: 1EdTech announced deprecation of LTI 1.1 (end-of-life 2025-06-30) — all tools must migrate to LTI 1.3. LTI 1.3 security model (OAuth 2.0 + JWT) is significantly more complex than LTI 1.1 (OAuth 1.0a). AI-powered LMS plugins are emerging that auto-configure LTI registrations. The xAPI + LRS pipeline is becoming the de facto standard for learning data interoperability, displacing proprietary event formats.
- **last_updated**: 2025-06
- **Decay risk**: low — LTI 1.3 is the stable standard for the foreseeable future; xAPI 2.0 is in development but backward-compatible.
- **失败模式**: JWT validation bugs → security vulnerabilities (token replay, impersonation); grade passback failures → teacher gradebook out of sync with adaptive system; user identity mismatch → student progress lost when accessing from different contexts; xAPI statements malformed → analytics pipeline breaks; not handling LMS downtime gracefully → students lose progress.

evidence: [T03-S021, T03-S022, T03-S023, T03-S030]

### 9. Compliance and Privacy Pipeline

- **Trigger**: Any collection/processing of student PII or educational records
- **Input**: Data inventory, applicable regulations, institutional policies
- **Output**: Compliance documentation, DPA agreements, technical safeguards

**入门 SOP**: 1. Conduct data inventory: catalog all student PII collected (name, email, interaction logs, assessment responses, device info), where it is stored, how it flows between services, and who has access. Create a data flow diagram. 2. Identify applicable regulations by market: US K-12 (FERPA + COPPA for <13); US higher ed (FERPA); EU (GDPR); China (个人信息保护法 + 双减); global (WCAG 2.1 AA for accessibility). 3. For FERPA: ensure the platform qualifies as a "school official" under the institution's annual notification, or obtain written consent. Implement "legitimate educational interest" access controls — only users with a need-to-know can access student records. 4. For COPPA (<13 users): implement verifiable parental consent before collecting data. The 2024 COPPA amendments require opt-in (not opt-out) for AI training and third-party sharing. 5. For GDPR: implement data subject rights (access, rectification, erasure, portability). Establish lawful basis (typically legitimate interest or contract for EdTech). Appoint DPO if processing data of EU students at scale. 6. Draft and sign Data Processing Agreements (DPAs) with all institutional customers: specify data ownership (institution owns), usage limitations (no advertising, no selling), retention/deletion schedule, breach notification timeline (72 hours for GDPR, varies by US state). 7. Implement technical safeguards: AES-256 encryption at rest, TLS 1.3 in transit, RBAC, audit logging, data minimization. For AI/LLM components: ensure student PII does not enter model training data (use RAG with ephemeral context, not fine-tuning on student data). 8. Conduct annual compliance audit; maintain SOC 2 Type II or equivalent certification.

- **资深路径**: 跳过 per-institution DPA negotiation — use standardized Student Data Privacy Consortium (SDPC) National Data Privacy Agreement that 200+ US districts accept. 优化 with automated compliance monitoring (privacy software that continuously scans for PII in logs, detects anomalous access patterns). 额外 implement privacy-by-design architecture: use stable internal identifiers instead of duplicating PII across services; implement differential privacy for aggregate analytics; build data residency support for multi-region deployment (EU data stays in EU).
- **AI 变化 (2024-2025)**: The 2024 COPPA amendments specifically target AI systems processing children's data — opt-in consent now required before any AI training on under-13 data. March 2025 DOE mandate requiring state-level FERPA certification is unprecedented enforcement. LLM-based tutoring creates new privacy challenges: student conversations may contain sensitive disclosures that must be handled carefully. Several states (California SOPIPA, Illinois SOPPA, Colorado) have enacted stricter student privacy laws beyond federal minimums.
- **last_updated**: 2025-06
- **Decay risk**: high — regulatory landscape is changing rapidly (new state laws, COPPA amendments, 个保法 enforcement); compliance is a moving target.
- **失败模式**: Data breach with student PII → regulatory fine + reputational damage + loss of institutional contracts; COPPA violation for under-13 users → FTC enforcement ($50K+ per violation); student data used for AI training without consent → regulatory and PR crisis; DPA not signed before data flows → institution liable; data retained beyond agreed period → compliance violation; student conversations with LLM tutor containing sensitive info not properly safeguarded.

evidence: [T03-S024, T03-S025, T03-S026, T03-S031]

### 10. AI Tutoring System Deployment

- **Trigger**: Decision to deploy LLM-based tutoring for a course or platform
- **Input**: Course content, pedagogical goals, target student population
- **Output**: Production LLM tutor with safety guardrails, monitoring, and human oversight

**入门 SOP**: 1. Define pedagogical approach: Socratic method (ask guiding questions, never give direct answers — Khanmigo model), direct instruction (explain then practice), or hybrid. Encode this in the system prompt. 2. Build the knowledge base: prepare course content as a RAG corpus — chunk textbook chapters, lecture notes, and solution manuals into retrievable segments. Use embeddings (text-embedding-3-large or equivalent) and a vector store (Pinecone, pgvector, Chroma). 3. Design the prompt architecture: system prompt (role definition, pedagogical rules, safety constraints) + RAG context (retrieved content) + conversation history + student model state (current KC mastery levels). 4. Implement safety guardrails — three layers: (a) input filters (block prompt injection, off-topic requests, inappropriate content); (b) output filters (block harmful content, check factual accuracy against RAG corpus, enforce pedagogical rules like "don't give direct answers"); (c) conversation monitoring (log all conversations, flag anomalies for human review). 5. Implement anti-cheating measures: detect and refuse to complete homework assignments directly; instead guide student through the reasoning process. 6. Build a feedback loop: domain experts review sampled conversations for factual accuracy, pedagogical correctness, tone, and safety compliance. 7. Pilot with a controlled cohort (e.g., one classroom) and measure learning outcomes vs. control group (no AI tutor). 8. Iterate on prompts, guardrails, and RAG retrieval based on pilot data. 9. Deploy to production with monitoring: track conversation length, student satisfaction, escalation rate to human teachers, and factual accuracy rate.

- **资深路径**: 跳过 generic RAG — implement curriculum-aware retrieval that uses the KC prerequisite graph to select contextually appropriate content for the student's current learning state. 优化 with fine-tuned models on domain-specific educational dialogues (e.g., math tutoring corpora) for better pedagogical quality. 额外 implement multi-turn assessment within tutoring conversations — the tutor simultaneously teaches and updates the student model, eliminating the need for separate assessments; build teacher override controls so educators can adjust the AI tutor's behavior per-student.
- **AI 变化 (2024-2025)**: Khanmigo's responsible AI framework (2024) established industry norms: conversation logging, content filtering, Socratic-only mode, and parental visibility. LPITutor (PMC 2025) demonstrates RAG + prompt engineering architecture for personalized tutoring. Multi-agent architectures are emerging: one LLM agent for tutoring, another for assessment, another for safety monitoring. Cost remains a barrier — GPT-4-level tutoring at $0.01-0.03 per conversation turn limits deployment in low-resource contexts. Open-source models (Llama 3, Mistral) are being fine-tuned for educational use to reduce costs.
- **last_updated**: 2025-06
- **Decay risk**: very high — LLM capabilities and best practices are evolving on a 3-6 month cycle; prompt architectures and safety frameworks from 2024 may be obsolete by late 2025.
- **失败模式**: LLM hallucination → student learns incorrect information (especially dangerous in STEM); too-helpful tutor gives direct answers → students don't learn; prompt injection → student manipulates tutor into inappropriate behavior; no human review pipeline → safety issues go undetected; RAG retrieval returns wrong content → tutor teaches off-topic material; high latency → student abandons conversation; cost overruns → unsustainable deployment.

evidence: [T03-S027, T03-S028, T03-S029]

---

## Phase 2 提炼提示

**工作流 → 心智模型映射**:
- WF1 (KC Analysis) → "Decompose → Validate → Iterate" loop — the core mental model is that learning objectives must be atomized before adaptation is possible; granularity is the fundamental design parameter
- WF2 (Item Calibration) → "Author → Pilot → Calibrate → Bank" pipeline — the psychometric quality gate; items without empirical parameters are guesses
- WF3 (Student Model) → "Data → Model → Inference → Update" cycle — the student is a probabilistic state machine; the model's job is to estimate which state the student is in
- WF4 (Adaptive Engine) → "Outer loop selects topics, inner loop provides feedback" — VanLehn's two-loop architecture is the dominant mental model for how adaptation actually works at runtime
- WF5 (Spaced Repetition) → "Memory decays exponentially; optimal review targets the forgetting threshold" — the memory model is the engine; everything else is scheduling
- WF6 (Analytics Dashboard) → "Data → Insight → Action" chain — if the dashboard doesn't suggest a concrete action, it's decoration
- WF7 (A/B Testing) → "Hypothesis → Randomize → Measure → Decide" scientific method applied to education — the discipline of not shipping without evidence
- WF8 (LTI/xAPI Integration) → "Standards are the contract between tools and platforms" — interoperability is non-negotiable for institutional adoption
- WF9 (Compliance) → "Map data flows → Apply regulations → Implement controls → Audit" — privacy is an architecture concern, not a checkbox
- WF10 (AI Tutoring) → "Pedagogy first, technology second" — the LLM is a delivery mechanism; the pedagogical design (Socratic vs. direct) determines outcomes

**入门 vs 资深的核心差异维度**:
- **Automation vs. Manual**: Beginners do each step manually (hand-author KCs, manually calibrate items, hand-tune thresholds); experts automate (AutoIRT, LLM-assisted KC decomposition, continuous calibration, automated Q-matrix refinement)
- **Fixed vs. Adaptive Pipelines**: Beginners run batch processes (calibrate once, train once, deploy); experts run continuous pipelines (online calibration, streaming model updates, adaptive experiment design)
- **Single-method vs. Ensemble**: Beginners use one model (BKT or DKT); experts ensemble multiple approaches and use the right model for each regime (BKT for cold-start, DKT for steady-state, LLM for zero-shot)
- **Compliance as Afterthought vs. Privacy-by-Design**: Beginners add privacy checks at the end; experts architect for privacy from the start (data minimization, differential privacy, purpose limitation)
- **Isolated Workflows vs. Integrated Loops**: Beginners treat each workflow independently; experts see them as a single system where the output of WF3 feeds WF4, which feeds WF6, which feeds WF7, which feeds back to WF1

**近 12 个月最大变化**:
- LLM-based tutoring (WF10) went from research prototype to production deployment (Khanmigo, LPITutor, dozens of startup products) — this is the single biggest shift in the field
- Training-free knowledge tracing (Thinking-KT, 2025) challenges the assumption that you need to train a model on student data — LLMs can estimate mastery from conversation history alone
- FSRS v5-v7 evolution made open-source spaced repetition algorithms definitively better than SM-2 for the first time, with benchmark proof on 500M+ reviews
- COPPA 2024 amendments + DOE March 2025 FERPA certification mandate dramatically raised the compliance bar for EdTech
- LTI 1.1 end-of-life (June 2025) forced all tool providers to migrate to LTI 1.3 — a major one-time integration cost
- AutoIRT and LLM-assisted item authoring are compressing the item development cycle from months to weeks

**工具 → 工作流对应**:
- pyBKT, DKT implementations, simpleKT → WF3 (Student Model Training)
- mirt (R), irtQ, AutoIRT, catsim → WF2 (Item Calibration)
- FSRS, HLR, Ebisu → WF5 (Spaced Repetition)
- CMU DataShop, learning curve tools → WF1 (KC Analysis) + WF3 (Student Model Validation)
- TAO, H5P, QTI tools → WF2 (Item Authoring)
- Watershed, Learning Locker, Ralph → WF6 (Analytics Dashboard)
- ltijs, pylti1p3, 1EdTech reference implementation → WF8 (LTI Integration)
- ASSISTments experimentation platform → WF7 (A/B Testing)
- ALEKS, MATHia, Squirrel AI → WF4 (Adaptive Engine) — commercial implementations of the full pipeline
- GPT-4, Claude, Llama 3 + RAG frameworks → WF10 (AI Tutoring)
- SDPC DPA templates, SOC 2 audit tools → WF9 (Compliance)
