Рус Eng During last 365 days Approved articles: 2075,   Articles in work: 306 Declined articles: 850 
Library
Articles and journals | Tariffs | Payments | Your profile

Back to contents

Reducing the complexity of the model of individual and group adaptive testing with multiple choice based on a fuzzy cognitive map
Kulikovskikh Ilona Markovna

PhD in Technical Science

Associate Professor, Department of Information Systems and Technologies, Academician Korolev Samara National Research University

443086, Russia, Samarskaya oblast', g. Samara, shosse Moskovskoe, 34

kulikovskikh.i@gmail.com
Prokhorov Sergej Antonovich

Doctor of Technical Science

Head of the Department of Information Systems and Technologies, Academician Korolev Samara National Research University

443086, Russia, Samarskaya oblast', g. Samara, shosse Moskovskoe, 34

sp.prokhorov@gmail.com

Abstract.

The subject of the study is adaptive testing with multiple choice questions. This type of testing allows you to implement a machine assessment of the level of knowledge of the participants is simple and accessible, eliminating errors of evaluating the results. However, the adaptive testing model includes a parameter describing the probability of guessing answers to test tasks, which depends on many factors: the difficulty of the task, the level of knowledge of the learner, the presence of a fine for trying to guess, and the degree to which the participant’s answers with a higher level of knowledge influence the opinions of other participants. in the conditions of individual and group testing. The need to explicitly set this parameter complicates the model and introduces uncertainty in the test results. The introduction of a fuzzy cognitive map, which determines the degree of "pure" and "partial" guessing in response to test tasks, reduces the complexity of the testing model as a result of excluding the explicit probabilistic parameter. Unlike the well-known definitions of a cognitive map, the proposed interpretation is based on models of individual and group adaptive testing with multiple choice. The results of computational experiments in real testing confirmed the effectiveness of the introduction of the map. It was found that fuzzy estimates of the responses of participants with a lower level of knowledge to more complex tasks are more consistent than estimates that require explicit assignment of the probability of "pure" guessing. The method of reducing the complexity of a testing model based on a fuzzy cognitive map can be used both in educational software systems and in intelligent systems and decision support systems that provide for testing with multiple choice.

Keywords: Bloom's taxonomy, interval-valued fuzzy set, cognitive modelling, knowledge assessment, pure guessing, partial guessing, collaborative learning, adaptive testing, cognitive map, logistic model

DOI:

10.7256/2454-0714.2018.4.28504

Article was received:

27-12-2018


Review date:

28-12-2018


Publish date:

29-12-2018


This article written in Russian. You can find full text of article in Russian here .

References
1.
Bessarabov N. A., Bondarenko A. V., Kondratenko T. N., Timofeev D. S. Algoritmicheskoe obespechenie adaptivnoi sistemy testirovaniya znanii // Programmnye produkty i sistemy. – 2016. – № 1(113). – S. 68-74.
2.
Kibzun A. I., Panarin S. I. Formirovanie integral'nogo reitinga s pomoshch'yu statisticheskoi obrabotki rezul'tatov testov // Avtomatika i telemekhanika. – 2012. – № 6. – S. 119-139.
3.
Kibzun A. I., Inozemtsev A. O. Otsenivanie urovnei slozhnosti testov na osnove metoda maksimal'nogo pravdopodobiya // Avtomatika i telemekhanika. – 2014. – № 4. – S. 20-37.
4.
Bloom B. S. (Ed.), Engelhart M. D., Furst E. J., Hill W. H., Krathwohl D. R. Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. – New York: David McKay, 1956. – 207 p.
5.
Anderson L. W. (Ed.), Krathwohl D. R. (Ed.), Airasian P. W., Cruikshank K.A., Mayer R. E., Pintrich P. R., Raths J., Wittrock M. C. A taxonomy for learning, teaching, and assessing: A revision of Bloom's Taxonomy of Educational Objectives (Complete edition). – New York: Longman, 2001. – 336 p.
6.
Mayer R. E. A taxonomy for computer-based assessment of problem-solving // Computers in Human Behaviour. – 2002. – 18. – pp. 623-632.
7.
Deutsch T., Herrmann K., Frese T., Sandholzer H. Implementing computer-based assessment-A web-based mock examination changes attitudes // Computers & Education. – 2012. – 58. – pp. 1068–1075.
8.
Kubinger K. D., Holocher-Ertl S., Reif M., Hohensinn C., Frebort M. On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format // International Journal of Selection and Assessment. – 2010. – 18(1). – pp. 111-115.
9.
Kuo C.-Y., Wu H.-K. Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessment in science // Computers & Education. – 2013. – 68. – pp. 388-403.
10.
Lesage E., Valcke M., Sabbe E. Scoring methods for multiple choice assessment in higher education-Is it still a matter of number right scoring or negative marking? // Studies in Educational Evaluation. – 2013. – 39. – pp. 188-193.
11.
Terzis V., Economides A. A. The acceptance and use of computer based assessment // Computers & Education. – 2011. – 56. – pp. 1032-1044.
12.
Thelwall M. Computer-based assessment: a versatile educational tool // Computers & Education. – 2000. – 34. – pp. 37-49.
13.
Wang T.-H. Developing an assessment-centered e-learning system for improving student learning effectiveness // Computers & Education. – 2014. – 73. – pp. 189-203.
14.
Bereby-Meyer Y., Meyer J., Budescu D. V. Decision making under internal uncertainty: the case of multiple-choice tests with different scoring rules // Acta Psychologica. – 2003. – 112. – pp. 207-220.
15.
Espinosa M. P., Gardeazabal J. Optimal correction for guessing in multiple-choice tests // Journal of Mathematical Psychology. – 2010. – 54. – pp. 415-425.
16.
Vanderoost J., Janssen R., Eggermont J., Callens R., De Laet T. Elimination testing with adapted scoring reduces guessing and anxiety in multiple-choice assessments, but does not increase grade average in comparison with negative marking // PLoS ONE. – 2018. – 13(10). – p. E0203931. DOI: 10.1371/journal.pone.0203931.
17.
Nurkova V. V., Gofman A. A. Zabyvanie: problema nalichiya sleda pamyati, ego dostupnosti i namerennogo kontrolya // Natsional'nyi psikhologicheskii zhurnal. – 2016. – № 3(23). – S. 64–74. DOI: 10.11621/npj.2016.0309.
18.
Nurkova V.V., Gofman A.A. Zabyvanie: problema nalichiya sleda pamyati, ego dostupnosti i namerennogo kontrolya // Natsional'nyi psikhologicheskii zhurnal. – 2016. – № 4(24). – S. 3–13. DOI: 10.11621/npj.2016.0401.
19.
Bjork E. L., Little J. L., Storm B. C. Multiple-choice testing as a desired difficulty in the classroom // Journal of Applied Research in Memory and Cognition. – 2014. – 3(3). – pp. 165-170.
20.
Bjork E. L., Soderstrom N. C., Little J. L. Can multiple-choice testing induce desirable difficulties? Evidence from the Laboratory and the Classroom // The American Journal of Psychology. – 2015. – 128(2). – pp. 229-239.
21.
Little J. L. The role of multiple-choice tests in increasing access to difficult-to-retrieve information // Journal of Cognitive Psychology. – 2018. – 30 (5-6). – pp. 520-531.
22.
Smith M. K., Wood W. B., Adams W. K., Wieman C., Knight J. K., Guild N. et al. Why peer discussion improves student performance on in-class concept questions // Science. – 2009. – 323. – pp. 122-124.
23.
Butler A., Roediger H. L. Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing // Memory & Cognition. – 2008. – 36(3). – pp. 604-616.
24.
Nickerson R. S.,Butler S. F., Carlin M. T. Knowledge assessment: Squeezing information from multiple-choice testing // Journal of Experimental Psychology: Applied. – 2015. – 21(2). – pp. 167-177.
25.
Nicol D. E-assessment by design: using multiple-choice tests to good effect // Journal of Further and Higher Education. – 2007. – 31(1). – pp. 53-64.
26.
Dehnad A., Nasser H., Hosseini A. F. A comparison between three and four option multiple choice questions // Procedia-Social and Behavioral Sciences. – 2014. – 98. – pp. 398-403.
27.
Lesage E., Valcke M., Sabbe E. Scoring methods for multiple choice assessment in higher education-Is it still a matter of number right scoring or negative marking? // Studies in Educational Evaluation. – 2013. – 39. – pp. 188–193.
28.
Kubinger K. D., Holocher-Ert S., Reif M., Hohensinn C., Frebort M. On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format//International Journal of Selection and Assessment. – 2010. – 18(1). – pp. 111-115.
29.
Macready G. B., Dayton C. M. The use of probabilistic models in the assessment of mastery // Journal of Educational Statistics. – 1977. – 2. – pp. 99-120.
30.
Van der Linden W. J. Forgetting, guessing, and mastery: The MacReady and Dayton models revisited and compared with a latent trait approach // Journal of Educational Statistics. – 1978. – 3(4). – pp. 305-317.
31.
Lord F. M., Novick M. R. Statistical theories of mental test scores. – Reading, MA: Addison Wesley, 1974. – 592 p.
32.
Rasch G. Probabilistic models for some intelligence and attainment tests. – Chicago: The University of Chicago Press, 1980. – 224 p.
33.
Kulinich A. A. Komp'yuternye sistemy modelirovaniya kognitivnykh kart: podkhody i metody // Problemy upravleniya. – 2014. – № 3. – S. 2-16.
34.
Kulinich A. A. Semioticheskie kognitivnye karty. Ch. 1. Kognitivnyi i semioticheskii podkhody v informatike i upravlenii // Problemy upravleniya. – 2016. – № 1. – S. 2-10.
35.
Kulinich A. A. Semioticheskie kognitivnye karty. Ch. 1. Osnovnye opredeleniya i algoritmy // Problemy upravleniya. – 2016. – № 2. – S. 24-40.
36.
Zhilov R. A. Optimizatsiya kognitivnoi karty dlya zadach prognozirovaniya // Kibernetika i programmirovanie. – 2015. – № 5. – S.128-135. DOI: 10.7256/2306-4196.2015.5.16592.
37.
Martyshenko S. N., Martyshenko N. S. Informatsionnaya tekhnologiya postroeniya kognitivnykh modelei // Programmnye sistemy i vychislitel'nye metody. – 2016. – № 4. – S. 362-374. DOI: 10.7256/2305-6061.2016.4.21456.
38.
Kitchin R. M. Cognitive maps: what are they and why study them? // Journal of Experimental Psychology. – 1994. – 14(1). – p. 1-19.
39.
Tolman E. C. Cognitive maps in rats and men // Psychological Review. – 1948. – 55(4). – pp. 189-208.
40.
Singpurwalla N. D., Booker J. M. Membership functions and probability measures of fuzzy sets // Journal of the American Statistical Association. – 2004. DOI: 10.1198/016214504000001196.
41.
Zadeh L. A. Toward a perception-based theory of probabilistic reasoning with imprecise probabilities // Journal of Statistical Planning and Inference. – 2002. – 105. – pp. 233-264.
42.
Zadeh L. A. Fuzzy possibilities // Information Processing & Management. – 1984. – 20(3). – pp. 363-372.
43.
Zadeh L. A. Fuzzy sets // Information and Control. – 1965. – 8. – pp. 338-353.
44.
Zadeh L. A. Fuzzy sets as a basis for a theory of possibility // Fuzzy Sets and Systems. – 1978. – 1. – pp. 3-28.
45.
Kulikovskikh I. M., Prokhorov S. A., Suchkova S. A., Matytsin E. V. Kompleksnaya sistema kollaborativnogo obucheniya na osnove nechetkikh modelei dlya opisaniya povedeniya sistem s chastichnym znaniem // Izvestiya SNTs RAN. – 2016. – t. 18, № 4(4). – S. 760-765.
46.
Zadeh L. A. Interval type-2 fuzzy logic systems: Theory and design // Inf. Sci. – 1971. – 3. – pp. 159-176.
47.
Bustince H., Fernandez J., Hagras H., Pagola M., Barrenechea E. Interval type-2 fuzzy sets are generalization of interval-valued fuzzy sets: Towards a wider view on their relationship // IEEE Tran. On Fuzzy Sets. – 2014. DOI: 10.1109/TFUZZ.2014.2362149