---
title: "Towards an Appropriate Query, Key, and Value Computation for"
description: "The authors propose SAINT (Separated Self-AttentIve Neural Knowledge Tracing), a Transformer… 23 references, cited by 7 papers here."
authors: ["Youngduck Choi", "Youngnam Lee", "Junghyun Cho", "Jineon Baek", "Byungsoo Kim", "Yeongmin Cha", "Dongmin Shin", "Chan Bae", "Jaewe Heo"]
year: 2020
venue: "Proceedings of the Seventh ACM Conference on Learning @ Scale"
source: "https://arxiv.org/abs/2002.07033"
doi: "10.1145/3386527.3405945"
arxiv: "2002.07033"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/choi-2020-appropriate-query-key-value-computation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/choi-2020-appropriate-query-key-value-computation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing

# Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing

**Authors:** Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae and Jaewe Heo · **Year:** 2020 · **Venue:** [Proceedings of the Seventh ACM Conference on Learning @ Scale](https://md.research.elamur.ai/free/en/venues/acm-conference-on-learning-scale-l-s/) ([L@S 2020](https://md.research.elamur.ai/free/en/venues/acm-conference-on-learning-scale-l-s/2020/)) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3386527.3405945) · [PDF](https://arxiv.org/pdf/2002.07033) · [DOI 10.1145/3386527.3405945](https://doi.org/10.1145/3386527.3405945) · [arXiv 2002.07033](https://arxiv.org/abs/2002.07033) · [OpenAlex](https://openalex.org/W3006521644)

**Citations:** 260 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/choi-2020-appropriate-query-key-value-computation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Knowledge tracing estimates how well a learner has mastered material from their past activity, typically as the chance of answering the next exercise correctly. The authors identify two weaknesses in earlier attention-based tracers (EKT, NPA, SAKT). These models use only one shallow attention layer, and they all build queries from exercises and keys/values from exercise-response interactions without testing alternatives. The authors propose SAINT (Separated Self-AttentIve Neural Knowledge Tracing), a Transformer encoder-decoder. The encoder applies stacked self-attention to exercise embeddings only. The decoder takes response embeddings and alternates self-attention with attention over the encoder output. Keeping the two streams apart lets the model stack many attention layers without losing accuracy. They tested SAINT on EdNet, a large dataset from the Santa mobile learning app. Compared with SAKT, SAINT raised AUC by 1.8%. It also outperformed several Transformer-based variants. The study shows that the choice of inputs for queries, keys and values matters a great deal in deep attention models for education.

## Key points
- **Dataset:** EdNet (from the Santa app) contains 627,347 users, 72,907,005 responses and 16,175 exercises, collected since 2016.
- **Best configuration:** SAINT with 4 stacked blocks and d_model = 512 reached ACC 0.7368 and AUC 0.7811. Among the compared architectures (SAINT, LTMTI, UTMTI, SSAKT), LTMTI came closest.
- **Attention patterns:** In the first decoder block, attention is sparse. In the last block, it is spread more evenly across past responses, which suggests deeper layers model richer relations.
- **Embedding ablation:** A simple response embedding (position plus correctness) performed slightly better than one that also used exercise category, timestamp and elapsed time.
- **Prior baselines:** The previous models compared against are EKT, NPA (Bi-LSTM with attention) and SAKT (single-layer Transformer).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 243 works · 251,774 citations
- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 94 works · 18,548 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 112 works · 14,009 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works

## Cite

Cite as: Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae and Jaewe Heo. 2020. Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing. Proceedings of the Seventh ACM Conference on Learning @ Scale.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{choi2020appropriate,
  title = {{Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing}},
  author = {Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae and Jaewe Heo},
  booktitle = {Proceedings of the Seventh ACM Conference on Learning @ Scale},
  year = {2020},
  doi = {10.1145/3386527.3405945},
  eprint = {2002.07033},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3386527.3405945}
}
```

- [← previous Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://md.research.elamur.ai/free/en/papers/allaway-2020-zero-shot-stance-detection-dataset/)
- [next → An Analysis of Simple Data Augmentation for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/dai-2020-analysis-simple-data-augmentation-named/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450 (2016).
2. Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2019. EdNet: A Large-Scale Hierarchical Dataset in Education. (2019). [find](https://scholar.google.com/scholar?q=Youngduck+Choi%2C+Youngnam+Lee%2C+Dongmin+Shin%2C+Junghyun+Cho%2C+Seoyon+Park%2C+Seewoo+Lee%2C+Jineon+Baek%2C+Chan+Bae%2C+Byungsoo+Kim%2C+and+Jaewe+Heo%2E+2019%2E+EdNet%3A+A+Large%2DScale+Hierarchical+Dataset+in+Education%2E+%2820)
3. Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253–278. [find](https://scholar.google.com/scholar?q=Albert+T+Corbett+and+John+R+Anderson%2E+1994%2E+Knowledge+tracing%3A+Modeling+the+acquisition+of+procedural+knowledge%2E+User+modeling+and+user%2Dadapted+interaction+4%2C+4+%281994%29%2C+253%E2%80%93278%2E)
4. Ryan SJ d Baker, Albert T Corbett, and Vincent Aleven. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In International conference on intelligent tutoring systems. Springer, 406–415. [find](https://scholar.google.com/scholar?q=Ryan+SJ+d+Baker%2C+Albert+T+Corbett%2C+and+Vincent+Aleven%2E+2008%2E+More+accurate+student+modeling+through+contextual+estimation+of+slip+and+guess+probabilities+in+bayesian+knowledge+tracing%2E+In+Internationa)
5. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249–256. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proceedings+of+the+thirteenth+international+conference+on+artificial+intelligence+a)
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
7. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. International World Wide Web Conferences Steering Committee, 173–182. [find](https://scholar.google.com/scholar?q=Xiangnan+He%2C+Lizi+Liao%2C+Hanwang+Zhang%2C+Liqiang+Nie%2C+Xia+Hu%2C+and+Tat%2DSeng+Chua%2E+2017%2E+Neural+collaborative+filtering%2E+In+Proceedings+of+the+26th+international+conference+on+world+wide+web%2E+Internationa)
8. Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, Guoping Hu, and others. 2019. EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction. IEEE Transactions on Knowledge and Data Engineering (2019). [find](https://scholar.google.com/scholar?q=Zhenya+Huang%2C+Yu+Yin%2C+Enhong+Chen%2C+Hui+Xiong%2C+Yu+Su%2C+Guoping+Hu%2C+and+others%2E+2019%2E+EKT%3A+Exercise%2Daware+Knowledge+Tracing+for+Student+Performance+Prediction%2E+IEEE+Transactions+on+Knowledge+and+Data+Eng)
9. Jussi Kasurinen and Uolevi Nikula. 2009. Estimating programming knowledge with Bayesian knowledge tracing. ACM SIGCSE Bulletin 41, 3 (2009), 313–317. [find](https://scholar.google.com/scholar?q=Jussi+Kasurinen+and+Uolevi+Nikula%2E+2009%2E+Estimating+programming+knowledge+with+Bayesian+knowledge+tracing%2E+ACM+SIGCSE+Bulletin+41%2C+3+%282009%29%2C+313%E2%80%93317%2E)
10. Diederik P Kingma and Jimmy Ba. 2014. [ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980 (2014).
11. Kangwook Lee, Jichan Chung, Yeongmin Cha, and Changho Suh. 2016. Machine Learning Approaches for Learning Analytics: Collaborative Filtering Or Regression With Experts?. In NIPS Workshop, Dec. 1–11. [find](https://scholar.google.com/scholar?q=Kangwook+Lee%2C+Jichan+Chung%2C+Yeongmin+Cha%2C+and+Changho+Suh%2E+2016%2E+Machine+Learning+Approaches+for+Learning+Analytics%3A+Collaborative+Filtering+Or+Regression+With+Experts%3F%2E+In+NIPS+Workshop%2C+Dec%2E+1%E2%80%9311%2E)
12. Youngnam Lee, Youngduck Choi, Junghyun Cho, Alexander R Fabbri, Hyunbin Loh, Chanyou Hwang, Yongku Lee, Sang-Wook Kim, and Dragomir Radev. 2019. Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess. arXiv preprint arXiv:1906.10910 (2019). — [arXiv:1906.10910](https://arxiv.org/abs/1906.10910) [find](https://scholar.google.com/scholar?q=Youngnam+Lee%2C+Youngduck+Choi%2C+Junghyun+Cho%2C+Alexander+R+Fabbri%2C+Hyunbin+Loh%2C+Chanyou+Hwang%2C+Yongku+Lee%2C+Sang%2DWook+Kim%2C+and+Dragomir+Radev%2E+2019%2E+Creating+A+Neural+Pedagogical+Agent+by+Jointly+Learning)
13. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. [Effective Approaches to Attention-based Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). arXiv preprint arXiv:1508.04025 (2015).
14. Shalini Pandey and George Karypis. 2019. [A Self-Attentive model for Knowledge Tracing](https://md.research.elamur.ai/free/en/papers/pandey-2019-self-attentive-model-knowledge-tracing/). arXiv preprint arXiv:1907.06837 (2019).
15. Zachary Pardos and Neil Heffernan. 2010a. Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm. In Educational Data Mining 2010. [find](https://scholar.google.com/scholar?q=Zachary+Pardos+and+Neil+Heffernan%2E+2010a%2E+Navigating+the+parameter+space+of+Bayesian+Knowledge+Tracing+models%3A+Visualizations+of+the+convergence+of+the+Expectation+Maximization+algorithm%2E+In+Education)
16. Zachary A Pardos, Yoav Bergner, Daniel T Seaton, and David E Pritchard. 2013. Adapting Bayesian Knowledge Tracing to a Massive Open Online Course in edX. EDM 13 (2013), 137–144. [find](https://scholar.google.com/scholar?q=Zachary+A+Pardos%2C+Yoav+Bergner%2C+Daniel+T+Seaton%2C+and+David+E+Pritchard%2E+2013%2E+Adapting+Bayesian+Knowledge+Tracing+to+a+Massive+Open+Online+Course+in+edX%2E+EDM+13+%282013%29%2C+137%E2%80%93144%2E)
17. Zachary A Pardos and Neil T Heffernan. 2010b. Modeling individualization in a bayesian networks implementation of knowledge tracing. In International Conference on User Modeling, Adaptation, and Personalization. Springer, 255–266. [find](https://scholar.google.com/scholar?q=Zachary+A+Pardos+and+Neil+T+Heffernan%2E+2010b%2E+Modeling+individualization+in+a+bayesian+networks+implementation+of+knowledge+tracing%2E+In+International+Conference+on+User+Modeling%2C+Adaptation%2C+and+Perso)
18. Yumeng Qiu, Yingmei Qi, Hanyuan Lu, Zachary A Pardos, and Neil T Heffernan. 2011. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing.. In EDM. 139–148. [find](https://scholar.google.com/scholar?q=Yumeng+Qiu%2C+Yingmei+Qi%2C+Hanyuan+Lu%2C+Zachary+A+Pardos%2C+and+Neil+T+Heffernan%2E+2011%2E+Does+Time+Matter%3F+Modeling+the+Effect+of+Time+with+Bayesian+Knowledge+Tracing%2E%2E+In+EDM%2E+139%E2%80%93148%2E)
19. Michael Sao Pedro, Ryan Baker, and Janice Gobert. 2013. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Educational Data Mining 2013. [find](https://scholar.google.com/scholar?q=Michael+Sao+Pedro%2C+Ryan+Baker%2C+and+Janice+Gobert%2E+2013%2E+Incorporating+scaffolding+and+tutor+context+into+bayesian+knowledge+tracing+to+predict+inquiry+skill+acquisition%2E+In+Educational+Data+Mining+201)
20. Nguyen Thai-Nghe, Lucas Drumond, Artus Krohn-Grimberghe, and Lars Schmidt-Thieme. 2010. Recommender system for predicting student performance. Procedia Computer Science 1, 2 (2010), 2811–2819. [find](https://scholar.google.com/scholar?q=Nguyen+Thai%2DNghe%2C+Lucas+Drumond%2C+Artus+Krohn%2DGrimberghe%2C+and+Lars+Schmidt%2DThieme%2E+2010%2E+Recommender+system+for+predicting+student+performance%2E+Procedia+Computer+Science+1%2C+2+%282010%29%2C+2811%E2%80%932819%2E)
21. Brett van De Sande. 2013. Properties of the bayesian knowledge tracing model. Journal of Educational Data Mining 5, 2 (2013), 1. [find](https://scholar.google.com/scholar?q=Brett+van+De+Sande%2E+2013%2E+Properties+of+the+bayesian+knowledge+tracing+model%2E+Journal+of+Educational+Data+Mining+5%2C+2+%282013%29%2C+1%2E)
22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems. 5998–6008.
23. Michael V Yudelson, Kenneth R Koedinger, and Geoffrey J Gordon. 2013. Individualized bayesian knowledge tracing models. In International conference on artificial intelligence in education. Springer, 171–180. [find](https://scholar.google.com/scholar?q=Michael+V+Yudelson%2C+Kenneth+R+Koedinger%2C+and+Geoffrey+J+Gordon%2E+2013%2E+Individualized+bayesian+knowledge+tracing+models%2E+In+International+conference+on+artificial+intelligence+in+education%2E+Springer%2C+1)

*Updated 3 November 2026*