A Mixed Methods Framework for Usability Profiling in Learning Environments: Integrating Adapted Heuristics and Machine Learning

Vista sencilla de ítem
dc.creatorOrtiz-Fajardo, Wilintonn Fidel
dc.creatorSolis-Pino, Andrés Felipe
dc.creatorCollazos, Cesar
dc.creatorConstain, Gustavo E.
dc.date2026-08-18
dc.date.accessioned2026-08-19T06:32:52Z
dc.descriptionTraditional usability evaluations of Learning Management Systems are often based on aggregate metrics that mask the heterogeneous experiences of diverse user populations, thus yielding limited actionable insights for personalized improvements. To address this significant gap, this research proposed a systematic six-phase methodological framework that integrates user-centered design principles, contextually adapted educational heuristics, and advanced machine learning techniques to deconstruct usability data and generate distinct data-driven user profiles. The methodology consisted of an empirical single-case study using a mixed-method approach to evaluate the implementation of a virtual Moodle environment for adult education in Comfacauca in Colombia. Data were collected from a heterogeneous sample of 50 students using the System Usability Scale questionnaire and the qualitative think-aloud protocol. These datasets were analyzed by combining descriptive statistics with Principal Component Analysis and K-means clustering. Although the platform achieved a strong aggregate SUS score of 73.9, analytical results enabled the successful identification of four distinct user profiles with unique technical needs, behavioral patterns, and cognitive challenges, which would have remained hidden with conventional single-score methods. For example, the framework revealed that perceptions of usability were more strongly correlated with the students' academic cycle than with their chronological age, directly challenging common demographic stereotypes. Finally, it is concluded that this comprehensive framework provides a reproducible methodological protocol for educational institutions to gain deeper, user-centered insights. This approach enables the transition from generic adjustments to the development of highly specific and differentiated support strategies, demonstrating the profound value of combining traditional usability methods with unsupervised clustering algorithms to transform general user feedback into actionable pedagogical intelligence.en-US
dc.descriptionLas evaluaciones de usabilidad tradicionales de los Sistemas de Gestión del Aprendizaje suelen basarse en métricas globales que enmascaran las experiencias heterogéneas de las diversas poblaciones de usuarios, proporcionando información útil limitada para realizar mejoras personalizadas. Para abordar esta brecha, esta investigación propuso un marco metodológico de seis fases que integró principios de diseño centrado en el usuario, heurísticas educativas adaptadas contextualmente y técnicas de aprendizaje automático para deconstruir datos de usabilidad y generar perfiles de usuario basados en datos. La metodología consistió en un estudio empírico de caso único que utilizó un enfoque de métodos mixtos para evaluar la implementación de un entorno virtual Moodle de educación de adultos en Comfacauca, Colombia. Los datos se recopilaron en una muestra heterogénea de 50 estudiantes utilizando la Escala de Usabilidad del Sistema y el protocolo cualitativo de pensamiento en voz alta. Estos conjuntos de datos se analizaron combinando estadística descriptiva con el Análisis de Componentes Principales y algoritmos de agrupamiento K-means. Aunque la plataforma obtuvo una buena puntuación global SUS de 73.9, los resultados lograron la identificación exitosa de cuatro perfiles de usuario distintos con necesidades técnicas, patrones de comportamiento y retos cognitivos, lo que habría permanecido invisible utilizando métodos convencionales de puntuación única. En esta línea, el marco reveló que la percepción de usabilidad estuvo más fuertemente correlacionada con el ciclo académico de los estudiantes que con su edad cronológica, desafiando estereotipos demográficos comunes. Finalmente, se concluye que este marco proporciona un protocolo metodológico reproducible para que las instituciones educativas obtengan conocimientos más profundos y centrados en el usuario. Este enfoque facilita el desarrollo de estrategias de apoyo específicas y diferenciadas, demostrando el valor de combinar métodos tradicionales de usabilidad con algoritmos de agrupamiento para transformar los comentarios generales en inteligencia pedagógica procesable.es-ES
dc.formatapplication/pdf
dc.identifierhttps://revistas.itm.edu.co/index.php/tecnologicas/article/view/3615
dc.identifier10.22430/22565337.3615
dc.identifier.urihttps://hdl.handle.net/20.500.12622/8206
dc.languageeng
dc.publisherInstituto Tecnológico Metropolitano (ITM)en-US
dc.relationhttps://revistas.itm.edu.co/index.php/tecnologicas/article/view/3615/4210
dc.relation/*ref*/I. N. Ainoutdinova, T. Tregubova, J. Ng, and V. A. Kopnov, “New roles and competencies of teachers in the ICT-mediated learning environment of Russian universities,” Educ. Sci. J., vol. 24, no. 1, pp. 191-221, Jan. 2022. https://doi.org/10.17853/1994-5639-2022-1-191-221
dc.relation/*ref*/H. Bagdi, S. Vardhan Pothabathula, L. Sharma, and H. P. Bulsara, “The global market upsurge in web traffic and revenues during the epidemic: an exploratory research of e-learning companies,” Int. J. Dev. Issues, vol. 22, no. 3, pp. 418-444, Nov. 2023. https://doi.org/10.1108/IJDI-06-2023-0147
dc.relation/*ref*/L. M. Lázaro Lorente, A. Ancheta Arrabal, and C. Pulido-Montes, “The Right to Education and ICT during COVID-19: An International Perspective,” Sustainability, vol. 12, no. 21, p. 9091, Oct. 2020. https://doi.org/10.3390/su12219091
dc.relation/*ref*/N. Kant, K. D. Prasad, and K. Anjali, “Selecting an appropriate learning management system in open and distance learning: a strategic approach,” Asian Assoc. Open Univ. J., vol. 16, no. 1, pp. 79-97, May. 2021. https://doi.org/10.1108/AAOUJ-09-2020-0075
dc.relation/*ref*/J. Cabero-Almenara, M. L. Arancibia, and A. Del Prete, “Technical and Didactic Knowledge of the Moodle LMS in Higher Education. Beyond Functional Use,” J. New Approaches Educ. Res., vol. 8, no. 1, pp. 25-33, Jan. 2019. https://doi.org/10.7821/naer.2019.1.327
dc.relation/*ref*/J. Ilham Saputro, L. Ferahma Sa’adah, Y. Syifa, K. Dara Ramadhanti, and M. Fikri Gojali, “Transforming Learning Experiences With Advanced Educational T echnology Solutions,” Int. Trans. Educ. Technol. ITEE, vol. 3, no. 2, pp. 114-124, Jan. 2025. https://doi.org/10.33050/itee.v3i2.721
dc.relation/*ref*/A. I. Almusharraf, “An Investigation of University Students’ Perceptions of Learning Management Systems: Insights for Enhancing Usability and Engagement,” Sustainability, vol. 16, no. 22, p. 10037, Nov. 2024. https://doi.org/10.3390/su162210037
dc.relation/*ref*/O. Güntem, and Y. Kılıç, “Efficiency and Sustainability in Online Education: An Evaluation of LMS Platforms and University Websites in Northern Cyprus,” Sustainability, vol. 17, no. 9, p. 4166, May. 2025. https://doi.org/10.3390/su17094166
dc.relation/*ref*/M. Astegher, P. Busetta, A. Gabbasov, M. Pedrotti, A. Perini, and A. Susi, “Specifying requirements for collection and analysis of online user feedback,” Requir. Eng., vol. 28, no. 1, pp. 75-96, Mar. 2023. https://doi.org/10.1007/s00766-022-00387-3
dc.relation/*ref*/M. Awad, K. Salameh, and E. L. Leiss, “Evaluating Learning Management System Usage at a Small University,” in Proceed. 2019 3rd Int. Conf. Information Syst. Data Mining, New York, NY, USA: ACM, 2019, pp. 98-102. https://doi.org/10.1145/3325917.3325929
dc.relation/*ref*/M. Gligora Marković, N. Kadoić, and B. Kovačić, “A Novel ANP-DEMATEL Framework for Multi-Criteria Decision-Making in Adaptive E-Learning Systems,” Mathematics, vol. 13, no. 22, p. 3714, Nov. 2025. https://doi.org/10.3390/math13223714
dc.relation/*ref*/D. Olugbade, O. A. Ojo, and A. E. Tolorunleke, “Challenges and Limitations of Moodle LMS in Handling Large-Scale Projects: West-African Universities Lecturers’ Perspective,” J. Educ. Technol. Instr., vol. 2, no. 2, pp. 47-66, Sep. 2023. https://www.researchgate.net/publication/374448084_Challenges_and_Limitations_of_Moodle_LMS_in_Handling_Large-Scale_Projects_West-African_Universities_Lecturers'_Perspective
dc.relation/*ref*/D. Dagger, A. O’Connor, S. Lawless, E. Walsh, and V. P. Wade, “Service-Oriented E-Learning Platforms: From Monolithic Systems to Flexible Services,” IEEE Internet Comput., vol. 11, no. 3, pp. 28-35, May. 2007. https://doi.org/10.1109/MIC.2007.70
dc.relation/*ref*/A. Liapis, E. Georgakakou, and A. Kameas, “Combining Heuristics and User Testing: Towards a Holistic Approach in User eXperience (UX) Evaluation of Open MOOC Platforms,” in Design, User Experience, and Usability. HCII 2025. Lecture Notes in Computer Science, vol. 15795, M. Schrepp, Ed., Cham, Swi: Springer Nature Switzerland, 2025, pp. 112-128. https://doi.org/10.1007/978-3-031-93224-3_8
dc.relation/*ref*/F. Pastrakis, M. Konstantakis, and G. Caridakis, “AI-Enhanced Modular Information Architecture for Cultural Heritage: Designing Cognitive-Efficient and User-Centered Experiences,” Information, vol. 17, no. 1, p. 92, Jan. 2026. https://doi.org/10.3390/info17010092
dc.relation/*ref*/P. Ngulube, and M. Masumbika Ncube, “Leveraging Learning Analytics to Improve the User Experience of Learning Management Systems in Higher Education Institutions,” Information, vol. 16, no. 5, p. 419, May. 2025. https://doi.org/10.3390/info16050419
dc.relation/*ref*/S. F. Mei Toh, P. Cruz Gonzalez, and K. N. K. Fong, “Usability of a wearable device for home-based upper limb telerehabilitation in persons with stroke: A mixed-methods study,” Digit. Health, vol. 9, p. 20552076231153737, Oct. 2023. https://doi.org/10.1177/20552076231153737
dc.relation/*ref*/A. Abid et al., “An evaluation framework and comparative analysis of the widely used learning management systems,” PLoS One, vol. 19, no. 12, p. e0311111, Dec. 2024. https://doi.org/10.1371/journal.pone.0311111
dc.relation/*ref*/W. Ortiz, A. F. Solis Pino, C. A. Collazos, and G. E. Constain, “Heuristic Evaluation for Improving Usability in LMS Platforms: A Study at CEVEC Virtual College,” presented at the XI Jornada Iberoam. Interacción Humano-Comput., Interacción Humano-Tecnología en la Era de la Inteligencia Artificial, Zacatecas, Mex, 2025. [Online]. Available: https://www.researchgate.net/publication/401701604_Heuristic_Evaluation_for_Improving_Usability_in_LMS_Platforms_AStudy_at_CEVEC_Virtual_College
dc.relation/*ref*/P. Vlachogianni, and N. Tselios, “Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review,” J. Res. Technol. Educ., vol. 54, no. 3, pp. 392-409, May. 2022. https://doi.org/10.1080/15391523.2020.1867938
dc.relation/*ref*/T. Doi, “Usability Textual Data Analysis: A Formulaic Coding Think-Aloud Protocol Method for Usability Evaluation,” Appl. Sci., vol. 11, no. 15, p. 7047, Jul. 2021. https://doi.org/10.3390/app11157047
dc.relation/*ref*/T. Hasrat, and K. Rosyadah, “Usability Factors as Antecedent and Consequence on Business Strategy and SERVQUAL: Nielsen & Mack Approach,” Gold. Ratio Mark. Appl. Psychol. Bus., vol. 1, no. 2, pp. 81-92, Jul. 2021. https://doi.org/10.52970/grmapb.v1i2.80
dc.relation/*ref*/A. F. Solis Pino, P. H. Ruiz, A. Mon, and C. A. Collazos, “ATLANTIS: A multidimensional maturity model for evaluating the adoption of the internet of things in industrial SMEs,” Internet Things, vol. 29, p. 101457, Jan. 2024. https://doi.org/10.1016/j.iot.2024.101457
dc.relation/*ref*/M. Benaida, “Developing and extending usability heuristics evaluation for user interface design via AHP,” Soft Comput., vol. 27, no. 14, pp. 9693-9707, Jul. 2023. https://doi.org/10.1007/s00500-022-07803-4
dc.relation/*ref*/A. P. Costa, F. N. de Sousa, A. Moreira, and D. N. de Souza, “Research through Design: Qualitative Analysis to Evaluate the Usability,” in Computer Supported Qualitative Research, A. P. Costa, L. P. Reis, F. Neri de Sousa, A. Moreira, and D. Lamas, Eds., Cham, Swi: Springer International Publishing, 2017, pp. 1-12. https://doi.org/10.1007/978-3-319-43271-7_1
dc.relation/*ref*/H. Carter-Roberts, R. Antbring, M. Angioi, and G. Pugh, “Usability testing of an e-learning resource designed to improve medical students’ physical activity prescription skills: a qualitative think-aloud study,” BMJ Open, vol. 11, no. 7, p. e042983, Jul. 2021. https://doi.org/10.1136/bmjopen-2020-042983
dc.relation/*ref*/J. Lorincz, K. Barišić, and V. Vlahović, “Usability Testing and the System Usability Scale Effectiveness Assessment on Different Sensing Devices of Prototype and Live Web System Counterpart,” Sensors, vol. 26, no. 2, p. 679, Jan. 2026. https://doi.org/10.3390/s26020679
dc.relation/*ref*/T. Pekkarinen, R. Uusitalo, and S. Kerr, “School tracking and intergenerational income mobility: Evidence from the Finnish comprehensive school reform,” J. Public Econ., vol. 93, no. 7-8, pp. 965-973, Aug. 2009. https://doi.org/10.1016/j.jpubeco.2009.04.006
dc.relation/*ref*/D. Peña, I. Vásquez Valencia, and A. F. Solis Pino, “Design and Implementation of an Incubator with PI Controller for Optimizing the Chicken Egg Incubation Process,” I T C- Res. Technol. Sci., vol. 1, no. 17, Dec. 2023. https://doi.org/10.57173/ritc.v1n17a18
dc.relation/*ref*/M. M. Alhammad, and A. M. Moreno, “Integrating user experience into Agile: an experience report on lean UX and Scrum,” in Proceed. ACM/IEEE 44th Int. Conf. Software Engin.: Software Engin. Educ. Train., New York, NY, USA: ACM, 2022, pp. 146-157. https://doi.org/10.1145/3510456.3514156
dc.relation/*ref*/ 
dc.relation/*ref*/M. A. Rodríguez-Ortiz, D. G. López Avalos, K. M. Ayala Quiles, and L. S. Gaytan-Lugo, “From Transcription to Empathy: Employing Artificial Intelligence Tools in User-Centered Design for an Online Assessment Platform,” Avanc. Interac. Hum.-Comput., vol. 9, no. 1, pp. 109-116, Nov. 2024. https://doi.org/10.47756/aihc.y9i1.155
dc.relation/*ref*/S. Hossain Munna, R. Hossain, and K. R. C. Saylo, “Digital Education Revolution: Evaluating LMS-based Learning and Traditional Approaches,” J. Innov. Technol. Converg., vol. 6, no. 2, pp. 21-40, Jun. 2024. https://doi.org/10.69478/JITC2024v6n002a03
dc.relation/*ref*/C. Hill, L. Du, M. Johnson, and B. D. McCullough, “Comparing programming languages for data analytics: Accuracy of estimation in Python and R,” WIREs Data Min. Knowl., vol. 14, no. 3, p. e1531, May. 2024. https://doi.org/10.1002/widm.1531
dc.relation/*ref*/A. F. Solis Pino, Z. Y. Delgado Espinosa, and E. V. Ramos Cabrera, “Characterization of the Rhizosphere Bacterial Microbiome and Coffee Bean Fermentation in the Castillo-Tambo and Bourbon Varieties in the Popayán-Colombia Plateau,” BMC Plant Biol., vol. 23, no. 1, p. 217, Apr. 2023. https://doi.org/10.1186/s12870-023-04182-2
dc.relation/*ref*/K. Wang, H. Ren, S. Yuan, X. Jiang, and P. Wang, “Exploring the diversity of dissolved organic matter (DOM) properties and sources in different functional areas of a typical macrophyte - derived lake combined with optical spectroscopy and FT-ICR MS analysis,” J. Environ. Sci., vol. 147, pp. 462-473, Jan. 2025. https://doi.org/10.1016/j.jes.2023.11.027
dc.relation/*ref*/K. Orfanou, N. Tselios, and C. Katsanos, “Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale,” Int. Rev. Res. Open Distrib. Learn., vol. 16, no. 2, Apr. 2015. https://doi.org/10.19173/irrodl.v16i2.1955
dc.relation/*ref*/A. K. Arora, J. Myers, T. Apramian, K. Kulasegaram, D. Bainbridge, and H. Seow, “Reliability and construct validation of the Blended Learning Usability Evaluation–Questionnaire with interprofessional clinicians in Canada: a methodological study,” J. Educ. Eval. Health Prof., vol. 22, p. 5, Jan. 2025. https://doi.org/10.3352/jeehp.2025.22.5
dc.relation/*ref*/O. Suria, “Assessing Financial Information System Usability Using System Usability Scale (SUS) and Usability Metric for User Experience-Lite (UMUX-Lite),” J. Teknol. Sist. Inf. Dan Apl., vol. 7, no. 2, pp. 537-547, Apr. 2024. https://doi.org/10.32493/jtsi.v7i2.38723
dc.relation/*ref*/A. Bangor, P. T. Kortum, and J. T. Miller, “An Empirical Evaluation of the System Usability Scale,” Int. J. Hum.-Comput. Interact., vol. 24, no. 6, pp. 574-594, Jul. 2008. https://doi.org/10.1080/10447310802205776
dc.relation/*ref*/G. Falloon, “From digital literacy to digital competence: the teacher digital competency (TDC) framework,” Educ. Technol. Res. Dev., vol. 68, no. 5, pp. 2449-2472, Oct. 2020. https://doi.org/10.1007/s11423-020-09767-4
dc.relation/*ref*/H. Tennakoon, J. M. Hansen, G. Saridakis, M. Samaratunga, and J. W. Hansen, “Drivers and Barriers of Social Sustainable Development and Growth of Online Higher Education: The Roles of Perceived Ease of Use and Perceived Usefulness,” Sustainability, vol. 15, no. 10, p. 8319, May. 2023. https://doi.org/10.3390/su15108319
dc.relation/*ref*/C. Keong Eric Loo, C. Y. Lim, and N. Athiyah Abdullah, “A Review of Usability Evaluation Framework for Tertiary Education Learning Management Systems,” IEEE Access, vol. 12, pp. 189941-189951, Dec. 2024. https://doi.org/10.1109/ACCESS.2024.3516000
dc.relation/*ref*/C. A. Halizha Chintya, and M. Fathurrahman, “Enhancing Interest and Achievement: Development of Smartbox Learning Media in Pancasila Education,” Edukasi J. Pendidik. Dan Pengajaran, vol. 12, no. 01, pp. 490-505, Jun. 2025. https://doi.org/10.19109/gfafdk27
dc.relation/*ref*/S. Kuswantono Wongsonadi, H. Hikmah, R. Dwi Lestari, and N. Hutama Putra, “Enhancing Eco-Facilitator Competencies through Interactive E-Workbook Development: A Study Using the ADDIE Model,” J. Pendidik. Indones., vol. 6, no. 2, pp. 1047-1065, Feb. 2025.
dc.relation/*ref*/T. Vanicek, and S. Popelka, “The Think-Aloud Method for Evaluating the Usability of a Regional Atlas,” ISPRS Int. J. Geo-Inf., vol. 12, no. 3, p. 95, Feb. 2023. https://doi.org/10.3390/ijgi12030095
dc.relation/*ref*/K. Hornbæk, “Current practice in measuring usability: Challenges to usability studies and research,” Int. J. Hum.-Comput. Stud., vol. 64, no. 2, pp. 79-102, Feb. 2006. https://doi.org/10.1016/j.ijhcs.2005.06.002
dc.relation/*ref*/J. LH. Birkland, “How older adult information and communication technology users are impacted by aging stereotypes: A multigenerational perspective,” New Media Soc., vol. 26, no. 7, pp. 3967-3988, Jul. 2024. https://doi.org/10.1177/14614448221108959
dc.relation/*ref*/
dc.rightsCopyright (c) 2026 TecnoLógicasen-US
dc.rightshttps://creativecommons.org/licenses/by-nc-sa/4.0en-US
dc.sourceTecnoLógicas; Vol. 29 No. 67 (2026)en-US
dc.sourceTecnoLógicas; Vol. 29 Núm. 67 (2026)es-ES
dc.source2256-5337
dc.source0123-7799
dc.subjectlearning environmenten-US
dc.subjectmachine learningen-US
dc.subjectusability evaluationen-US
dc.subjectuser-centered designen-US
dc.subjectuser profilesen-US
dc.subjectentorno de aprendizajees-ES
dc.subjectaprendizaje automáticoes-ES
dc.subjectevaluación de usabilidades-ES
dc.subjectdiseño centrado en el usuarioes-ES
dc.subjectperfiles de usuarioes-ES
dc.titleA Mixed Methods Framework for Usability Profiling in Learning Environments: Integrating Adapted Heuristics and Machine Learningen-US
dc.titleUn framework de métodos mixtos para el perfilado de usabilidad en entornos de aprendizaje: integrando heurísticas adaptadas y aprendizaje automáticoes-ES
dc.typeinfo:eu-repo/semantics/article
dc.typeinfo:eu-repo/semantics/publishedVersion
dc.typeResearch Papersen-US
dc.typeArtículos de investigaciónes-ES

Archivos

Colecciones

Revista TecnoLógicas - Cosecha Portal de Revistas