Validating graded vocabulary lists through targeted crowdsourcing

Graded word lists are important to language learners, as they give clues as to when certain words should be taught. They can also be of interest to researchers and teachers who might want to analyze student essays with regard to vocabulary distribution. While there has been quite substantial work for English, Swedish is much more underrepresented. There have been previous efforts at creating graded vocabulary lists for Swedish for language learners (Volodina & Johansson Kokkinakis, 2012; François et al., 2016; Volodina et al., 2016). However, most of these efforts were concentrated on extracting distributions of words over levels from graded essays and textbooks targeting certain levels. Thus, for a given word, it could be said that it occurred X times at level A1 and Y times at level A2, etc. There have also been recent works that try mapping these distributions over levels to single target levels (Gala et al., 2014; Alfter et al., 2016). However, none of these mapping approaches have been thoroughly tested for validity.

We investigate how we can use an online language learning platform and different exercise types to validate the correctness of the automatically graded vocabulary lists. The word lists in question cover receptive vocabulary knowledge, derived from a textbook corpus, and productive vocabulary knowledge, derived from graded student essays. We surmise that by embedding words of uncertain level into exercises targeting a certain sub-population of learners (e.g. intermediate/B1 learners), conclusions can be drawn about the “real” level of the vocabulary items. If learners that are judged to be of intermediate level have difficulties with words that our automatic assignment labeled as appropriate for intermediate level learners, we can assume that those words are probably of a higher level than predicted. We present exercise types that target productive vocabulary knowledge as well as exercise types that target receptive vocabulary knowledge and the current experiments to establish a methodology that would allow us to draw conclusions based on learner reponses to these test items.


David Alfter, Yuri Bizzoni, Anders Agebjörn, Elena Volodina, Ildikó Pilán. 2016. From Distributions to Labels: A Lexical Proficiency Analysis using Learner Corpora. Proceedings of the workshop on NLP4CALL&LA. NEALT Proceedings Series / Linköping Electronic Conference Proceedings.

Thomas François, Elena Volodina, Ildikó Pilán, Anaïs Tack. 2016. SVALex: a CEFR-graded lexical resource for Swedish foreign and second language learners. Proceedings of LREC 2016, Slovenia.

Núria Gala, Thomas François, Delphine Bernhard, Cédric Fairon. 2014. Un modèle pour prédir la complexité lexicale et graduer les mots. Actes de la conférence Traitement Automatique des Langues Naturelles (TALN 2014). Marseille.

Elena Volodina and Sofie Johansson Kokkinakis. 2012. Introducing Swedish Kelly-list, a new lexical e-resource for Swedish. LREC 2012, Turkey.

Elena Volodina, Ildikó Pilán, Lorena Llozhi, Baptiste Degryse, Thomas François. 2016. SweLLex: second language learners’ productive vocabulary. Proceedings of the workshop on NLP4CALL&LA. NEALT Proceedings Series / Linköping Electronic Conference Proceedings