Received: Noviembre 30, 2023
Accepted: Marzo 08, 2024
Available: Abril 24, 2024

Resumen

El pensamiento computacional (PC) es una nueva forma de alfabetización y se considera como una competencia clave para los ciudadanos de la era actual. Es un constructo compuesto que tiene relación con la resolución de problemas, el pensamiento matemático, el pensamiento crítico, la creatividad y la comunicación. La investigación sobre la evaluación del PC se encuentra en consolidación, sin embargo, se evidencia ausencia de agrupación sistemática de instrumentos de medición del PC en diferentes niveles educativos. El objetivo de esta revisión consistió en identificar los instrumentos usados como herramientas para medir el PC, las habilidades evaluadas y las propiedades psicométricas de los instrumentos. Esta revisión sistemática presentó el análisis de 52 artículos encontrados del 2012 al 2022. Los resultados de la revisión demostraron un crecimiento significativo en las publicaciones relacionadas con el diseño y la validación de instrumentos de medición del PC en los últimos años. Se encontró que más del 80 % de los instrumentos presentaron evidencia de validez y confiabilidad, destacando la validez de contenido, la validez de constructo y la consistencia interna. Así mismo, en algunos instrumentos se consideraron la evaluación de habilidades afectivas, sociales y actitudes, lo cual enriquecía la valoración de las habilidades cognitivas. Sin embargo, se evidenció la ausencia de los países de Centro y Sur América en los artículos analizados sobre esta temática, al igual que la escasez de instrumentos dirigidos a la primera infancia y a los docentes. Estos hallazgos resaltan la necesidad de continuar investigando el PC desde la perspectiva de la evaluación en poblaciones específicas.

1. INTRODUCTION

In recent decades, the scientific and educational communities have stressed the importance of incorporating Computational Thinking (CT) into curricula across all levels of education. Nonetheless, given its status as an emerging field, there is still no consensus on its definition and practical application. Consequently, a variety of approaches have been adopted to integrate it in students’ learning process. This lack of a standardized definition for CT makes it challenging to design methods and tools for its evaluation [1]. Moreover, the rapid advancement of information and communication technologies underscores the need for 21st-century individuals to develop digital skills [2], [3], which brings about changes in how people think, act, communicate, and solve problems.

From a conceptual standpoint, various definitions have been put forth for CT. For instance, [4] serves as a starting point, defining it as the process of applying basic computer science principles to solve problems, design systems, and understand human behavior. [5], for his part, analyzed the different definitions of CT that have been proposed from the generic, operational, psychological-cognitive, and educational-curricular perspectives. In their literature review, [6] suggested classifying the definitions based on two approaches. The first approach is concerned with the relationship between computational concepts and programming, where authors [7]–[9] stand out. The second approach pertains to the set of competencies that students should develop, encompassing domain-specific knowledge and problem-solving skills. In this latter approach, the authors highlight the proposals of the International Society for Technology in Education (ISTE) and the Computer Science Teachers Association (CSTA) [10], along with [11] and [12].

Several initiatives have been developed to integrate CT into curricula, as well as tools for its accurate and reliable assessment. These tools include questionnaires [13]–[15], task-based tests [16], coding activities [17], and observation. Nevertheless, for widespread application across various educational levels, it is imperative to enhance the measurement of this construct using instruments with psychometric properties.

This systematic review was motivated by the need to measure CT and identify the instruments that have been designed for its assessment. Specifically, the goal is to analyze a set of bibliometric indicators and variables of interest, such as the type of instrument, number of items, target population, sample size, evidence of pilot testing, identification of skills/competencies, theoretical foundations, and psychometric properties.

1.1 Literature review

An examination of previous studies on the use of CT assessment tools provided valuable insights into the current landscape of this field. The search yielded 15 reviews, classified into different categories: scoping reviews [18]–[20], systematic or bibliometric mappings [21], [22], systematic reviews [6], [23]–[29], and meta-analyses [30], [31]. The analysis of these reviews underscored the need to identify the instruments employed for measuring CT, their psychometric properties, and the various variables associated with CT.

The retrieved review articles contributed to shedding light on research related to CT assessment. For instance, a scoping review of CT assessments in higher education [19] unveiled empirical studies focusing on CT assessments in post-secondary education. The majority of the analyzed instruments sought to measure CT skills by combining various dimensions, including concepts, practices, and perspectives. Among the skills frequently evaluated in these studies were algorithmic thinking, problem-solving, data handling, logic, and abstraction. However, it is worth noting that only four of these instruments provided sufficient evidence of their reliability and validity.

In another scoping review of empirical research on recent CT assessments [18], the authors classified features related to graphical or block-based programming, web-based simulation, robotics-based games, tests, and scales. Most studies in this review adopted a quasi-experimental approach, with only a few providing evidence of their validity. This review highlighted the need to carry out assessments aimed at different levels of higher-order thinking skills.

In their systematic review [6], evaluated 96 articles, considering variables such as educational level, subject matter domain, educational setting, and assessment tool. The findings emphasized the need for more assessments targeting high school students, college students, and teachers, in addition to evidence of the validity and reliability of the instruments. [25], for their part, analyzed 64 studies on CT measurement, identifying the psychometric properties of instruments primarily aimed at determining levels and measuring skills.

In the identified scoping reviews, mapping reviews, and meta-analyses, Scopus, ScienceDirect, ERIC, and Web of Science (WOS) were the most frequently consulted sources. As for the target population, Figure 1 shows that the most commonly selected population was students in K-12 educational settings. No reviews targeting teachers were found in the analysis.

Figure 1. Target population in the retrieved reviews
Source: Own work.

Out of the 15 reviews, the one conducted by [22] included the largest number of sample articles (321 in total), while [23] had the smallest sample (15 articles). The sample sizes of the other reviews ranged from 17 to 101 articles. Figure 2 illustrates the number of articles included in the 15 reviews.

Figure 2. Target population in the retrieved reviews
Source: Own work.

Additionally, Table 1 lists the titles of the review articles, along with the country where the research was conducted.

2. METHODOLOGY

This systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [33]. Such a protocol involves the following steps: (a) defining the research questions, objectives, and study variables (bibliometric indicators and variables of interest); (b) conducting a literature search (definition of search strings, period of analysis, inclusion and exclusion criteria, and sources of information, and study selection); and (c) identifying relevant articles.

2.1 Research questions, objectives, and variables

The following research questions were proposed for this systematic review:

All these questions serve to identify the tools that have been used for CT assessment, their psychometric properties, and the evaluated skills. The proposed variables were divided into two categories: (i) bibliometric indicators, encompassing title, source of information, publication year, country, language, authors, journal, quartile, and the Scientific Journal Rank (SJR) and Journal Citation Reports (JCR) indices; and (ii) variables of interest, including type of instrument, number of items, age of target population, evaluated skills/competencies, theoretical foundations, authors, sample size, pilot testing, and method for determining instrument validity and reliability.

2.2 Literature search

To conduct the search, the following eight search strings were formulated, incorporating key concepts such as computational thinking, measurement, and instruments, while adhering to the syntax required by the employed databases:

"Pensamiento Computacional" AND medición

"Computational Thinking" AND measuring

"Computational thinking" + "measuring instruments"

"Computational thinking" + "measurement"

"Computational thinking" + "measure instruments"

"Computational thinking" + "measurement tool"

"Computational thinking" AND ("measur* instruments" OR "measur* tool*")

"Computational thinking" AND (“assess” OR “validity” OR “reliability” OR “test” OR “scale”)

The search spanned from 2012 to 2022 because, as indicated by [34], this is when CT started to consolidate as a construct. For the search, five sources of information were consulted: ScienceDirect, EBSCO Discovery, Scopus, WOS, and Springer.

Regarding exclusion and inclusion criteria, only research articles and reviews were considered for analysis, while publications in book formats, posters, conference proceedings, or articles that did not employ a specific instrument for measuring CT were excluded.

2.3 Identified articles

Initially, the search yielded 439 articles. After removing duplicates, 204 articles remained. Following further screening for relevance, 115 articles were retained. Finally, by applying exclusion criteria, a total of 52 articles were selected for the systematic review. Figure 3 provides a summary of the articles identified at each stage of the search process, which was conducted following the PRISMA statement.

Figure 3. Systematic review flowchart
Source: Adapted from the PRISMA statement [35].

3. RESULTS AND DISCUSSION

3.1 Analysis of bibliometric indicators

3.1.1 Consulted databases

The majority of the articles (approximately 63.4 %) were found in Scopus. Figure 4 shows the number of articles retrieved from each consulted database, with several appearing in multiple sources.

Figure 4. Consulted databases
Source: Own work.

3.1.2 Title keywords

According to the analysis, computational thinking was the most prevalent term in the titles of the examined articles, often accompanied by valid, scale, evaluate, and test, all of which allude to important features of the measurement instruments.

3.1.3 Publication year

As mentioned earlier, the search spanned from 2012 to 2022. Remarkably, none of the five sources yielded publications related to instrument construction before 2017. Figure 5 depicts the increase in the number of publications dedicated to CT measurement instruments throughout the analyzed period.

Figure 5. Publication year of articles into CT measurement instruments
Source: Own work.

3.1.4 Country where the research was conducted

Türkiye was the country with the highest number of articles—ten in total—followed by China and the United States, with eight and seven articles, respectively. Only one study was carried out in Latin America, specifically in Venezuela. Figura 6 displays the distribution of articles by country.

Figure 6. Distribution of articles by country
Source: Own work.

3.1.5 Language

English was the most prevalent language, with 90 % of the articles being written in this language. Spanish accounted for 4 %, Turkish 4 %, and Japanese 2 %.

3.1.6 Authors

Table 2 presents the most prominent authors based on the number of published articles.

Also, one important aspect considered in the analysis was the most cited authors in the analyzed articles (see Figure 7). Prominent authors include Brennan and Resnick, Selby and Woollard, the International Society for Technology in Education (ISTE), the Computer Science Teacher Association (CSTA), Román et al., and Korkmaz et al. The latter authors are notable references because the instruments they proposed—the Computational Thinking Test (CTt) and the Computational Thinking Scale (CTS)—are frequently employed for CT measurement.

Figure 7. Most cited authors
Source: Own work.

3.1.7 Journal, quartile, and JCR indices

In the analysis, two impact indicators evaluating the excellence of published content were employed. JCR, on the one hand, primarily focuses on citation counts, providing the impact factor and quartile of a journal. SJR, on the other hand, considers the quality, relative importance of citations, and quartile of a journal. According to the findings, the Journal of Educational Computing Research and Education and Information Technologies stood out as the most productive journals, with six and four publications, respectively. Regarding the two indices and quartiles, 21 % of the journals had no classification in any of the indices. Information on each journal is provided in Table 3. The most prominent journals, ranked by the number of CT-related publications, are listed in [22] .

Figure 8 summarizes the quartiles assigned to the journals in which the articles were published. A total of 40 different journals were identified, of which 47.5 % have been classified in a JCR quartile and 65 % have been classified with the SJR indicator.

Figura 8. Journals classified into quartiles
Source: Own work.

4. DISCUSSION

This systematic literature review aimed to identify studies that have introduced instruments for measuring CT, as well as bibliometric variables and other variables of interest to delve deeper into this object of study.

A total of 52 research papers and 15 meta-analyses, mapping reviews, and systematic literature reviews were selected. Four studies are noteworthy ([6], [18], [19], [25]) because they make evident what supports, adds value, and justifies this literature review: the need for an analysis of the psychometric properties of those instruments.

For this purpose, it was necessary to establish what methods have been used to determine their validity and reliability. This process can be addressed from the perspective of the Classical Test Theory (CTT) and the Item Response Theory (IRT). CTT is based on methods that evaluate the quality of tests by measuring the internal consistency and validity of the content, criterion, and construct. On the other hand, the IRT offers a more advanced approach as it considers the individual characteristics of the items and participants, enabling a more accurate estimation of the skills under evaluation and a more sensitive assessment of performance. Integrating both theories allows for a more comprehensive and reliable evaluation of tests, facilitating decision-making in various educational and professional contexts. This review includes several articles that refer to adaptations of two instruments: The Computational Thinking Test (CTt) and the self-report Computational Thinking Scale (CTS). All the adaptations of the CTS [46] and CTt [37] , [38] have shown evidence of psychometric properties.

Considering the authors of the 52 articles and those most cited within them, Marcos Román-Gonzaléz and Özgen Korkmaz were found to be at the top of both lists, demonstrating their extensive research experience in CT assessment.

The protocol for this review included six questions that can be used to delve deeper into this discussion:

Skills, attitudes, and perceptions are dimensions widely used to measure CT. Some studies [18]–[22], [27] focus on measuring CT through computational practices and concepts. All these reviews cite Brennan and Resnick’s [7] curriculum guide as a foundational resource for CT. Other studies [6], [18], [19], [23], [29] use self-report scales to analyze students’ perceptions and preferences. Only one systematic literature review [24] defines abstraction from a multi-dimensional perspective.

5. CONCLUSIONS

This study reviewed 52 research articles about CT assessment and measurement published between 2012 and 2022 in academic journals. Additionally, it analyzed scoping reviews, systematic mapping reviews, and literature reviews, which revealed (a) the interest in consolidating the evidence on CT assessment and (b) research gaps for this review. Consequently, this literature review was conducted to learn about the psychometric properties of CT assessment and measurement instruments, as well as CT-related variables.

This systematic review implemented a process that ensures the repeatability of the review protocol. The research questions helped to define the limits of the bibliometric variables and variables of interest. The bibliometric variables indicate that the number of articles on CT measurement instruments has increased since 2019. Most documents on CT measurement have been published in Türkiye, the US, and China. Based on the JCR index, 27.5 % of the articles were published in Q1 journals; and based on the SJR index, 47.5 % were featured in Q1 outlets. There is no evidence of CT measurement instruments in Colombia. Regarding key authors, Brennan and Resnick, as well as Selby and Woollard, are commonly cited due to their widely recognized CT curriculum designs. Marcos Román González and Özgen Korkmaz stand out for their numerous publications and the international adaptation of their instruments. The Computational Thinking Scale (CTS) has been adapted and psychometrically validated in Europe and Asia, while the Computational Thinking Test (CTt) has been adapted and its validity has been established in the same regions. However, no adaptations of these instruments to Latin American countries were identified. As the instruments were mostly applied to high school and college students, future research should address other populations, such as young children or adults.

The results of this review highlight the diverse range of CT skills that can be evaluated. Among these skills (that have a multidimensional origin), algorithmic thinking, cognitive skills, and problem-solving capabilities are the most common. Computational capabilities are also widely assessed, especially concepts that are directly related to computer programming, such as sequences, conditionals, loops, and events. It should be noted that abstraction has been commonly evaluated across all populations, but there is little scientific evidence of a rigorous evaluation of this construct. Only Ezeamuzie et al. have formally operationalized this skill.

This review revealed a variety of instruments to measure CT—with scales being the most frequently used format. This suggests that CT should be assessed in a comprehensive manner by addressing a wide range of associated skills, concepts, attitudes, and procedures. Most reviewed instruments demonstrated both validity and reliability, with content and construct validity, as well as internal consistency, being the predominant criteria. The statistical methods most commonly employed to analyze these properties are correlation, Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), and Cronbach’s alpha.

This literature review makes a contribution to future studies by demonstrating the progress made in CT assessment through the use of measurement instruments with strong psychometric properties. In conclusion, this review accomplished its objective, i.e., it identified the tools that have been used to measure CT, along with their psychometric properties and the skills they assess.

6. ACKNOWLEDGMENTS AND FUNDING

The authors thank the Ph.D. program in Education Sciences at Universidad del Quindío, which provided funding for Milena Corrales Álvarez and Lina Marcela Ocampo to complete their doctoral studies.

They would also like to thank Universidad del Quindío for funding the research project (Code 1187 of the Call for Proposals No. 14 of 2022).

CONFLICTS OF INTEREST

The authors declare that there is no conflict of interest.

AUTHOR CONTRIBUTIONS

Milena Corrales-Álvarez: Methodology, Conceptualization, Investigation, and Writing - Review & Editing.

Lina Marcela Ocampo: Methodology, Conceptualization, Investigation, and Writing - Review & Editing.

Sergio Augusto Cardona-Torres: Conceptualization, Investigation, and Writing - Review & Editing.

7. REFERENCES

• arrow_upward [1] H. Arranz de la Fuente, and A. Pérez García, “Evaluación del pensamiento computacional en educación primaria,” Rev. Interuniv. Investig. en Tecnol. Educ. Educ., no. 3, pp. 25–39, 2017. http://dx.doi.org/10.6018/riite/2017/267411
• arrow_upward [2] C. Fadel, M. Bialik, and B. Trilling, Educación en cuatro dimensiones: las competencias que los estudiantes necesitan para su realización, Santiago, Chile: Grafhika Impresores, 2016. [Online]. Available: https://www.educarchile.cl/recursos-para-el-aula/educacion-en-cuatro-dimensiones
• arrow_upward [3] C. L. Scott, “El futuro del aprendizaje 2 ¿Qué tipo de aprendizaje se necesita para el siglo XXI?,” Organización de las Naciones Unidas para la Educación, la Ciencia y la Cultura, París, Rep. ERF No. 14, 2015. [Online]. Available: https://repositorio.minedu.gob.pe/bitstream/handle/20.500.12799/4661/El%20futuro%20del%20aprendizaje%202%20Qu%c3%a9%20 tipo%20de%20aprendizaje%20se%20necesita%20para%20el%20siglo%20XXI.pdf?sequence=1&isAllowed=y
• arrow_upward [4] J. M. Wing, “Computational thinking and thinking about computing,” Philos. Trans. R. Soc. A Math. Phys. Eng. Sci., vol. 366, no. 1881, pp. 3717–3725, Oct. 2008. https://doi.org/10.1098/rsta.2008.0118
• arrow_upward [5] M. Román-González, “Codigoalfabetización y pensamiento computacional en educación primaria y secundaria: validación de un instrumento y evaluación de programas,” Ph.D. dissertation, Escuela Internacional de Doctorado, Universidad Nacional de Educación a Distancia, España, 2016. [Online]. Available: http://e-spacio.uned.es/fez/view/tesisuned:Educacion-Mroman
• arrow_upward [6]   X. Tang, Y. Yin, Q. Lin, R. Hadad, and X. Zhai, “Assessing computational thinking: A systematic review of empirical studies,” Comput. Educ., vol. 148, p. 103798, Apr. 2020. https://doi.org/10.1016/j.compedu.2019.103798
• arrow_upward[7] K. Brennan, and M. Resnick, “New frameworks for studying and assessing the development of computational thinking,” in Proceedings of the 2012 annual meeting of the American educational research association, Vancouver, Canada, Apr. 2012, p. 25.[Online]. Available: https://scratched.gse.harvard.edu/ct/files/AERA2012.pdf
• arrow_upward [8] D. Weintrop et al., “Defining Computational Thinking for Mathematics and Science Classrooms,” J. Sci. Educ. Technol., vol. 25, pp. 127–147, Feb. 2016. https://doi.org/10.1007/s10956-015-9581-5
• arrow_upward [9]  L. Werner, J. Denner, and S. Campe, “Children programming games: A strategy for measuring computational learning,” ACM Trans. Comput. Educ., vol. 14, no. 4, pp. 1-22, Dec. 2014. https://doi.org/10.1145/2677091
• arrow_upward [10] ISTE, and CSTA, “Operational definition of computational thinking for K-12 education.,” National Science Foundation, 2011. [Online]. Available: https://cdn.iste.org/www-root/Computational_Thinking_Operational_Definition_ISTE.pdf
• arrow_upward [11] C. Selby, and J. Woollard, “Computational thinking: the developing definition,” University of Southampton, Inglaterra, Rep. Soton356481, Oct. 2013. [Online]. Available: http://eprints.soton.ac.uk/id/eprint/356481
• arrow_upward [12] A. Yadav, C. Mayfield, N. Zhou, S. Hambrusch, and J. T. Korb, “Computational thinking in elementary and secondary teacher education,” ACM Trans. Comput. Educ., vol. 14, no. 1, pp. 1–16, Mar. 2014. https://doi.org/10.1145/2576872
• arrow_upward [13] E. Relkin, L. de Ruiter, and M. U. Bers, “TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education,” J. Sci. Educ. Technol., vol. 29, no. 4, pp. 482–498, Aug. 2020. https://doi.org/10.1007/s10956-020-09831-x
• arrow_upward [14] E. Relkin, and M. Bers, “TechCheck-K: A measure of computational thinking for kindergarten children,” in 2021 IEEE Global Engineering Education Conference, Vienna, Austria, 2021, pp. 1696-1702. https://doi.org/10.1109/EDUCON46332.2021.9453926
• arrow_upward [15] L. El-Hamamsy et al., “Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners’ CT test (BCTt) and the competent CT test (cCTt),” Front. Psychol., vol. 13, Dec. 2022. https://doi.org/10.3389/fpsyg.2022.1082659
• arrow_upward [16] L. Wang, F. Geng, X. Hao, D. Shi, T. Wang, and Y. Li, “Measuring coding ability in young children: relations to computational thinking, creative thinking, and working memory,” Curr. Psychol., vol. 42, no. 10, pp. 8039-8050, Apr. 2021. https://doi.org/10.1007/s12144-021-02085-9
• arrow_upward [17]K. Kanaki, and M. Kalogiannakis, “Assessing Algorithmic Thinking Skills in Relation to Gender in Early Childhood,” Educ. Process Int. J., vol. 11, no. 2, pp. 44–59, Jun. 2022. https://doi.org/10.22521/edupij.2022.112.3
• arrow_upward [18] M. Cutumisu, C. Adams, and C. Lu, “A Scoping Review of Empirical Research on Recent Computational Thinking Assessments,” J. Sci. Educ. Technol., vol. 28, no. 6, pp. 651–676, Dec. 2019. https://doi.org/10.1007/s10956-019-09799-3
• arrow_upward [19] C. Lu, R. Macdonald, B. Odell, and V. Kokhan, “A scoping review of computational thinking assessments in higher education,” J. Comput. High. Educ., vol. 34, no. 2, pp. 416–461, Aug. 2022. https://doi.org/10.1007/s12528-021-09305-y
• arrow_upward [20] K. I. McCormick, and J. A. Hall, “Computational thinking learning experiences, outcomes, and research in preschool settings: a scoping review of literature,” Educ. Inf. Technol., vol. 27, no. 3, pp. 3777–3812, Apr. 2022. https://doi.org/10.1007/s10639-021-10765-z
• arrow_upward [21] N. Da Cruz Alves, C. Gresse Von Wangenheim, and J. C. R. Hauck, “Approaches to assess computational thinking competences based on code analysis in K-12 education: A systematic mapping study,” Informatics Educ., vol. 18, no. 1, pp. 17–39, Apr. 2019. https://doi.org/10.15388/infedu.2019.02
• arrow_upward [22] M. Tekdal, “Trends and development in research on computational thinking,” Educ. Inf. Technol., vol. 26, no. 5, pp. 6499–6529, Sep. 2021. https://doi.org/10.1007/s10639-021-10617-w
• arrow_upward [23] E. Bakala, A. Gerosa, J. P. Hourcade, and G. Tejera, “Preschool children, robots, and computational thinking: A systematic review,” Int. J. Child-Computer Interact., vol. 29, p. 100337, Sep. 2021. https://doi.org/10.1016/j.ijcci.2021.100337
• arrow_upward [24] N. O. Ezeamuzie, J. S. C. Leung, and F. S. T. Ting, “Unleashing the Potential of Abstraction From Cloud of Computational Thinking: A Systematic Review of Literature,” J. Educ. Comput. Res., vol. 60, no. 4, pp. 877–905, Dec. 2022. https://doi.org/10.1177/07356331211055379
• arrow_upward [25] H. İ. Haseki, and U. Ilic, “An Investigation of the Data Collection Instruments Developed to Measure Computational Thinking,” Informatics Educ., vol. 18, no. 2, pp. 297–319, Oct. 2019. https://doi.org/10.15388/infedu.2019.14
• arrow_upward [26] P. Kakavas, and F. C. Ugolini, “Computational thinking in primary education : a systematic literature review,” vol. 11, no. 2. Dec. 2019. https://doi.org/10.2478/rem-2019-0023
• arrow_upward[27] N. O. Ezeamuzie, and J. S. C. Leung, “Computational Thinking Through an Empirical Lens: A Systematic Review of Literature,” J. Educ. Comput. Res., vol. 60, no. 2, pp. 481–511, Apr. 2022. https://doi.org/10.1177/07356331211033158
• arrow_upward [28] E. Taslibeyaz, E. Kursun, and S. Karaman, “How to Develop Computational Thinking: A Systematic Review of Empirical Studies,” Informatics Educ., vol. 19, no. 4, pp. 701–719, Dec. 2020. https://doi.org/10.15388/infedu.2020.30
• arrow_upward [29] C. Tikva, and E. Tambouris, “Mapping Computational Thinking through Programming in K-12 Education: A Conceptual Model based on a Systematic Literature Review,” Comput. Educ., vol. 162, p. 104083, Mar. 2020. https://doi.org/10.1016/j.compedu.2020.104083
• arrow_upward [30] H. Lei, M. M. Chiu, F. Li, X. Wang, and G. Ya-Jing, “Computational thinking and academic achievement: A meta-analysis among students,” Child. Youth Serv. Rev., vol. 118, p. 105439, Nov. 2020. https://doi.org/10.1016/j.childyouth.2020.105439
• arrow_upward [31] L. Sun, L. Hu, and D. Zhou, “Which way of design programming activities is more effective to promote K‐12 students’ computational thinking skills? A meta‐analysis,” J. Comput. Assist. Learn., vol. 37, no. 4, pp. 1048–1062, Aug. 2021. https://doi.org/10.1111/jcal.12545
• arrow_upward [32] L. Sun, L. Hu, and D. Zhou, “The bidirectional predictions between primary school students’ STEM and language academic achievements and computational thinking: The moderating role of gender,” Think. Ski. Creat., vol. 44, p. 101043, Jun. 2022. https://doi.org/10.1016/j.tsc.2022.101043
• arrow_upward [33] A. C. Tricco et al., “PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation,” Ann. Intern. Med., vol. 169, no. 7, pp. 467–473, Sep. 2018. https://doi.org/10.7326/M18-0850
• arrow_upward [34] J. M. Wing, “Computational thinking,” Commun. ACM, vol. 49, no. 3, pp. 33–35, Mar. 2006. https://dl.acm.org/doi/pdf/10.1145/1118178.1118215
• arrow_upward [35] G. Urrútia, and X. Bonfill, “PRISMA declaration: A proposal to improve the publication of systematic reviews and meta-analyses,” Medicina Clínica, vol. 135, no. 11, pp. 507–511, Oct. 2010. https://doi.org/10.1016/j.medcli.2010.01.015
• arrow_upward [36] Y. Li, S. Xu, and J. Liu, “Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students,” J. Pacific Rim Psychol., vol. 15, Aug. 2021. https://doi.org/10.1177/18344909211010240
• arrow_upward [37] M. Román-González, J. C. Pérez-González, J. Moreno-León, and G. Robles, “Can computational talent be detected? Predictive validity of the Computational Thinking Test,” Int. J. Child-Computer Interact., vol. 18, pp. 47–58, Nov. 2018. https://doi.org/10.1016/j.ijcci.2018.06.004
• arrow_upward [38] M. Román-González, J. C. Pérez-González, and C. Jiménez-Fernández, “Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test,” Comput. Human Behav., vol. 72, pp. 678–691, Jul. 2017. https://doi.org/10.1016/j.chb.2016.08.047
• arrow_upward [39] K. Lee, and J. Cho, “Computational Thinking Evaluation Tool Development for Early Childhood Software Education,” JOIV Int. J. Informatics Vis., vol. 5, no. 3, p. 313, Sep. 2021. https://doi.org/10.30630/joiv.5.3.672
• arrow_upward [40] Y. Lee, and J. Cho, “Toward Developing a Real-World Computational Thinking Test Tool from Existing Models,” Int. J. Recent Technol. Eng., vol. 8, no. 2S6, pp. 389–393, Jul. 2019. https://doi.org/10.35940/ijrte.B1073.0782S619
• arrow_upward [41] S. Sovey, K. Osman, and M. E. E. M. Matore, “Gender differential item functioning analysis in measuring computational thinking disposition among secondary school students,” Front. Psychiatry, vol. 13, Nov. 2022. https://doi.org/10.3389/fpsyt.2022.1022304
• arrow_upward [42] S. Sovey, K. Osman, and M. E. E. Mohd-Matore, “Exploratory and Confirmatory Factor Analysis for Disposition Levels of Computational Thinking Instrument Among Secondary School Students,” Eur. J. Educ. Res., vol. 11, no. 2, pp. 639–652, Apr. 2022. https://doi.org/10.12973/eu-jer.11.2.639
• arrow_upward [43] S. C. Kong, and Y. Q. Wang, “Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts,” Comput. Human Behav., vol. 122, p. 106836, Sep. 2021. https://doi.org/10.1016/j.chb.2021.106836
• arrow_upward [44] K. Siu-Cheung, and M. Lai, “Effects of a teacher development program on teachers’ knowledge and collaborative engagement, and students’ achievement in computational thinking concepts,” Br. J. Educ. Technol., vol. 54, no. 2, pp. 489–512, Mar. 2023. https://doi.org/10.1111/bjet.13256
• arrow_upward [45] L. El-Hamamsy, M. Zapata-Cáceres, E. M. Barroso, F. Mondada, J. D. Zufferey, and B. Bruno, “The Competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary School,” J. Educ. Comput. Res., vol. 60, no. 7, pp. 1818–1866, May. 2022. https://doi.org/10.1177/07356331221081753
• arrow_upward [46] Ö. Korkmaz, R. Çakir, and M. Y. Özden, “A validity and reliability study of the computational thinking scales (CTS),” Comput. Human Behav., vol. 72, pp. 558–569, Jul. 2017. https://doi.org/10.1016/j.chb.2017.01.005
• arrow_upward [47] Ö. Korkmaz, and X. Bai, “Adapting Computational Thinking Scale (CTS) for Chinese High School Students and Their Thinking Scale Skills Level,” Particip. Educ. Res., vol. 6, no. 1, pp. 10–26, Jun. 2019. https://doi.org/10.17275/per.19.2.6.1
• arrow_upward [48] E. Çoban, and Ö. Korkmaz, “An alternative approach for measuring computational thinking: Performance-based platform,” Think. Ski. Creat., vol. 42, p. 100929, Dec. 2021. https://doi.org/10.1016/j.tsc.2021.100929
• arrow_upward [49] M. Lafuente Martínez, O. Lévêque, I. Benítez, C. Hardebolle, and J. D. Zufferey, “Assessing Computational Thinking: Development and Validation of the Algorithmic Thinking Test for Adults,” J. Educ. Comput. Res., vol. 60, no. 6, pp. 1436–1463, Feb. 2022. https://doi.org/10.1177/07356331211057819
• arrow_upward [50] J. Guggemos, S. Seufert, and M. Román-González, “Computational Thinking Assessment – Towards More Vivid Interpretations,” Technol. Knowl. Learn., vol. 28, no. 2, pp. 539–568, Jun. 2023. https://doi.org/10.1007/s10758-021-09587-2
• arrow_upward [51] D. E. Sondakh, K. Osman, and S. Zainudin, “A Proposal for Holistic Assessment of Computational Thinking for Undergraduate: Content Validity,” Eur. J. Educ. Res., vol. 9, no. 1, pp. 33–50, Jan. 2020. https://doi.org/10.12973/eu-jer.9.1.33
• arrow_upward [52] D. E. Sondakh, K. Osman, and S. Zainudin, “A Pilot Study of an Instrument to Assess Undergraduates’ Computational thinking Proficiency,” Int. J. Adv. Comput. Sci. Appl., vol. 11, no. 11, 2020. https://doi.org/10.14569/IJACSA.2020.0111134
• arrow_upward [53] E. Polat, S. Hopcan, S. Kucuk, and B. Sisman, “A comprehensive assessment of secondary school students’ computational thinking skills,” Br. J. Educ. Technol., vol. 52, no. 5, pp. 1965–1980, Sep. 2021. https://doi.org/10.1111/bjet.13092
• arrow_upward [54] L. Sun, L. Hu, D. Zhou, and W. Yang, “Evaluation and developmental suggestions on undergraduates’ computational thinking: a theoretical framework guided by Marzano’s new taxonomy,” Interact. Learn. Environ., vol. 31, no. 10, pp. 6588–6610, Mar. 2023. https://doi.org/10.1080/10494820.2022.2042311
• arrow_upward [55] M. Junpho, A. Songsriwittaya, and P. Tep, “Reliability and Construct Validity of Computational Thinking Scale for Junior High School Students: Thai Adaptation,” Int. J. Learn. Teach. Educ. Res., vol. 21, no. 9, pp. 154–173, Sep. 2022. https://doi.org/10.26803/ijlter.21.9.9
• arrow_upward [56] H. Nakahara et al., “Development of the Japanese Version of the Computational Thinking Scales for First-Year University Students in Humanities,” Japan J. Educ. Technol., vol. 46, no. 1, 2022. https://doi.org/10.15077/jjet.45089
• arrow_upward [57] L. Mujica De Statzewitch, “Evaluación del Desarrollo del Pensamiento Computacional en Estudiantes de Educación Primaria y Media General.,” Rev. Digit. del Dr. en Educ. la Univ. Cent. Venez., vol. 7, no. 13, pp. 35–56, Jan-Jun. 2021. http://saber.ucv.ve/ojs/index.php/rev_arete/article/view/21327/144814487622
• arrow_upward [58] İ. Çetin, T. Otu, and A. Oktaç, “Adaption of the Computational Thinking Test into Turkish,” Turkish J. Comput. Math. Educ., vol. 11, no. 2, May. 2020. https://doi.org/10.16949/turkbilmat.643709
• arrow_upward [59] C. Shiau-Wei, L. Chee-Kit, and B. Sumintono, “Assessing computational thinking abilities among Singapore secondary students: a Rasch model measurement analysis,” J. Comput. Educ., vol. 8, no. 2, pp. 213–236, Jun. 2021. https://doi.org/10.1007/s40692-020-00177-2
• arrow_upward [60] A. Dolmaci, and N. E. Akhan, “The Development of Computational Thinking Skills Scale: Validity and Reliability Study,” İnsan ve Toplum Bilim. Araştırmaları Derg., vol. 9, no. 3, pp. 1970–1991, Sep. 2020. https://doi.org/10.15869/itobiad.698736
• arrow_upward [61] B. Ertugrul-Akyol, “Development of Computational Thinking Scale: Validity and Reliability Study,” Int. J. Educ. Methodol., vol. 5, no. 3, pp. 421–432, Aug. 2019. https://doi.org/10.12973/ijem.5.3.421
• arrow_upward [62] K. Bati, and M. İkbal Yetişir, “Examination of Turkish Middle School STEM Teachers’ Knowledge about Computational Thinking and Views Regarding Information and Communications Technology,” Comput. Sch., vol. 38, no. 1, pp. 57–73, Mar. 2021. https://doi.org/10.1080/07380569.2021.1882206
• arrow_upward [63] D. C. Boulden, A. Rachmatullah, K. M. Oliver, and E. Wiebe, “Measuring in-service teacher self-efficacy for teaching computational thinking: development and validation of the T-STEM CT,” Educ. Inf. Technol., vol. 26, no. 4, pp. 4663–4689, Jul. 2021. https://doi.org/10.1007/s10639-021-10487-2
• arrow_upward [64] P. J. Rich, R. A. Larsen, and S. L. Mason, “Measuring teacher beliefs about coding and computational thinking,” J. Res. Technol. Educ., vol. 53, no. 3, pp. 296–316, Jul. 2021. https://doi.org/10.1080/15391523.2020.1771232
• arrow_upward [65] J. Sung, “Assessing young Korean children’s computational thinking: A validation study of two measurements,” Educ. Inf. Technol., vol. 27, no. 9, pp. 12969–12997, Nov. 2022. https://doi.org/10.1007/s10639-022-11137-x
• arrow_upward [66] M. Zapata-Cáceres, “Enseñanza, evaluación y análisis de habilidades de pensamiento computacional en etapas tempranas,” Ph.D. dissertation, Universidad Rey Juan Carlos, Madrid, España, 2022. [Online]. Available: https://burjcdigital.urjc.es/handle/10115/19965?show=full
• arrow_upward [67] S. Grover, and R. D. Pea, “"Systems of Assessments" for Deeper Learning of Computational Thinking in K-12,” in Annual Meeting of the American Educational Research Association, Chicago, IL, USA, Apr. 15-20, 2015. [Online]. Available: https://www.researchgate.net/publication/275771253
• arrow_upward [68] K. Siu-Cheung, and H. Abelson, Eds., Computational Thinking Education, 1st. ed. Gateway East, Singapore: Springer Singapore, 2019. https://doi.org/10.1007/978-981-13-6528-7
• arrow_upward [69]  M. del M. Sánchez Vera, “Computational Thinking in Educational Environments: An Approach from Educational Technology,” Research in Education and Learning Innovation Archives, no. 23, p. 24-39, Jul-Dec. 2019. https://doi.org/10.7203/realia.23.15635