Personalized learning paths recommendation system with collaborative filtering and content-based approaches
- University of Economics and Law, Ho Chi Minh City, Vietnam
- Vietnam National University, Ho Chi Minh City, Vietnam
Abstract
Recommender systems have undergone a transformative evolution, reshaping user interactions across diverse domains. Notably, the emphasis on personalized learning paths has grown significantly in education. This research paper delves into the performance evaluation of User-based Collaborative Filtering and Content-based recommendation techniques to develop innovative recommender systems explicitly tailored to Information Systems students. By integrating the primary dataset collection rooted within the Knowledge - Skill - Attitude framework for students in the Faculty of Information Systems at the University of Economics and Law, this study assesses how effectively these two separate models develop personalized recommendation systems. Furthermore, the empirical evaluation of two distinct models, the Collaborative Filtering and Content-Based approach, across key metrics such as Precision, Recall, and F1-Score, provides a comprehensive view of their effectiveness in generating a Recommendation System for the University of Economics and Law. Findings reveal that the Collaborative Filtering approach excels in Precision, achieving a perfect score. At the same time, the Content-based technique demonstrates superior recall capabilities, suggesting its potential to cater to diverse educational needs. This paper also highlights the transformative role of recommendation systems in higher education, particularly in enhancing student engagement through personalized learning experiences and aligning curricula with industry requirements. Recognizing the limitations inherent in deploying either model independently, future research should propose a hybrid approach that combines the strengths of both Collaborative Filtering and Content-based methods, aiming to mitigate the existing drawbacks of the distinct model. The findings provide actionable insights for students, universities, and businesses to enhance educational content and career development tools and pave the way for future research on hybrid recommendation methodologies, which promise a more tailored and efficient learning experience for learners.
Introduction
The rapid evolution of technology and job markets in Information Technology (IT) dramatically transformed the career development landscape. Students must adhere to a continuous learning philosophy to remain competitive in the ever-changing Information Systems (IS).
Navigating the vast array of learning options in IS poses a significant challenge, as students must discern which paths will be most effective for their career advancement. Wan and Zhang 1 argued that while beneficial, the abundance of online resources can lead to confusion and decision paralysis, underscoring the need for tailored guidance. Zhou et al. 2 noted that this context necessitates a focused approach toward developing Recommender Systems (RS) for Personalized Learning Paths (PLP), catering specifically to the IS domain, as a vital tool for navigating the extensive digital learning environment. Niknam and Thulasiraman3 stated that the emerging need for these systems is driven by the increasing obsolescence of traditional career planning and educational methods in the IS sector in the face of novel technological breakthroughs and market dynamics.
This challenge was further amplified by the necessity to align learning choices with the rapidly evolving IS industry demands. Moreover, a study by Joseph et al. 4 emphasized the critical connection between these learning opportunities and long-term career goals in IS, demanding a careful balance between immediate skill acquisition and future career objectives. Chen et al.5 recognized the gap between the skills imparted by traditional education and those demanded in the workplace there is a pressing need for recommendation systems that align learning choices with industry requirements.
This research aims to evaluate the efficacy of user-user collaborative filtering and content-based techniques in developing innovative recommender systems. With this objective, the research methodology integrates two distinct primary approaches: user-user collaborative filtering and content-based techniques to provide personalized learning pathways aligned with the Knowledge - Skill - Attitude (KSA) framework, focusing on students in the Faculty of Information Systems (FIS) at the University of Economics and Law (UEL). The objective is to develop customized systems for IS students, ensuring a seamless integration of techniques for enhanced learning experiences.
This paper begins with a literature review to establish context, followed by a detailed exposition of the methodology. Subsequent sections present and discuss the research findings, exploring their implications for continuous learning in the IS domain. The paper concludes with recommendations for future research initiatives.
Literature review
RS has dynamically transformed user interaction across various domains, including education, where its role in shaping PLP was increasingly recognized. To contextualize this research within this evolving landscape, Lu et al. 6 emphasized the transformative potential of RS in education, highlighting the need for high-quality, instructive reviews of current trends. These systems enhanced user experience and engagement by predicting user preferences through various algorithms.
Marappan and Saraswatikaniga7 argued that the Collaborative Filtering (CF) approach is the most established and widely utilized method. This research underscored its fundamental reliance on the intricate dynamics of user-item interactions. This method's strength in identifying patterns among users to suggest personalized content is pivotal. CF leveraged similarities between users and items to generate personalized recommendations. Abdi et al. 8 underlined the effectiveness of Matrix Factorization in CF, particularly for large datasets, despite acknowledging the hurdles, such as data sparsity, that can affect recommendation quality. While the CF approach was celebrated for its ability to tailor recommendations based on user-item interactions, critics argued that it may not sufficiently capture the full spectrum of user preferences, especially in diverse educational contexts. As noted by another study 9, concerns about data sparsity and privacy suggested limitations in CF's applicability without robust data handling and privacy safeguards. Furthermore, the reliance on existing user interactions could narrow learning opportunities, overlooking emerging or interdisciplinary content that could enrich the learner's experience 10.
On the other hand, Content-Based Recommender Systems (CBRS) recommend items based on a user’s historical item-rating data. Murugan et al. 11 noted CBRS's prevalence in research-paper recommendations but pointed out the ambiguity in their effectiveness compared to CF. This uncertainty, often stemming from the challenges in accurately mapping user preferences to content features, was particularly relevant to this investigation. In educational settings, where the content is diverse and often complex, ensuring that recommendations are relevant and conducive to learning objectives is a significant challenge [12, p. 72]. Lops et al. 13 also added that ensuring diversity and serendipity in recommendations remains challenging for CBRS. Another paper further contributed to this discussion by addressing the need for diversity and serendipity in CB recommendations 7, 14. In educational RS, it is essential that the system not only caters to the known preferences of learners but also exposes them to a broader range of learning materials that could spark new interests and learning paths, a point that this research considers.
Discussing previous work on RS in PLP, Kirkwood and Price15 discussed previous work on RS in PLP and indicated a gap between theory and practice in the field of Technology-Enhanced Learning (TEL). This underdevelopment in RS for PLP has been an area our research directly addresses. The author also stressed the need for more research on RS assessment, pointing out the potential discrepancies between these systems' perceived and actual effectiveness 13. This is considered a deeper evaluation of RS, an aspect that is central to this study.
Implementing RS in PLP within educational settings, notably higher education, presents a unique set of challenges and opportunities. A study 16 showed that balancing customized learning experiences with curriculum frameworks and job requirements remains challenging. Huu et al.17 stated that while RS can build highly personalized learning paths, aligning these with expected learning outcomes and job descriptions was a tension this paper seeks to explore. Their observation revealed a discrepancy between theoretical advancements and practical applications in this field. The scarcity of RS in PLP highlights a significant gap where potential benefits are yet to be fully harnessed in real-world educational settings18.
In conclusion, the potential of RS in education to enhance PLP has been clarified, yet various challenges need to be addressed. As a result, this creates a need for the adoption of data-driven research to assess the effectiveness of RS in educational contexts. This approach is also vital for substantiating the potential of RS in improving educational outcomes19. Future research would focus on developing RS that are not only technologically advanced but also pedagogically sound, effectively bridging the gap between user needs and the evolving requirements of the modern workforce.
Methodology
The current study aims to assess the performance of personalized RS by conducting a comparative analysis of two distinct models, including the CF and CB. Figure 1 outlines five-specific steps of the research framework for this project, beginning with the data-storing phase to model evaluation in a structured workflow.
Research Framework (Source: Authors)
Dataset Description
Figure 2 can be considered a comprehensive compilation of data that provides insights into the competency needs of various IT job titles. It includes 7,000 entries and 13 columns outlining essential IT skills and competencies required for each unique job title. Each skill was quantified using advanced Natural Language Processing (NLP) techniques to rank each skill based on market relevance and demand to ensure a comprehensive resource for understanding IT skill requirements. The research also utilized Bloom's Taxonomy to ensure a focused, all-inclusive approach to ascertain the proposed IT skill requirements.
Skill Dataset (Source: Authors)
The Knowledge Dataset includes 77 courses covering fundamental programming principles in specialized fields such as machine learning and cybersecurity. The corresponding 'learning_outcomes' column highlights the practicality and significance of the course material in addressing real-world challenges and job responsibilities to establish a clear correlation between academic pursuits and the professional skill sets essential to excel in the IS field.
The Attitude dataset was established to highlight vital personal qualities. The dataset includes 'job_title' and 'attitude' columns that link attributes like problem-solving, adaptability, teamwork, and analytical thinking to specific IT roles. The 'attitude' columns are derived from an analysis that emphasizes the top three qualities of each job title. This underscores the importance of continuous learning and collaboration in navigating the evolving technological landscape and executing complex projects.
Algorithm Implementation
User-User-Based Collaborative Filtering
This RS utilizes User-Based CF to produce personalized recommendations by analyzing mutual preferences and user interactions. Heap et al. 20 said that this approach adopts the Cosine imilarity - a widely recognized metric calculating the cosine of the angle between two non-zero vectors in a multi-dimensional space, to determine the similarity between user and job profiles. The formula is described as follows:
Here, 𝐴 and 𝐵 are user interaction vectors. For example, if User X and Job Y have interacted with skills represented by vectors [3, 2, 0, 5] and [1, 0, 4, 4], respectively, the cosine similarity was calculated based on these vectors, providing a quantifiable measure of their preference alignment. Initially, a skill-rating matrix was established, capturing the interactions and preferences of all users within the system. Subsequently, similarity scores were computed for each user pair using the cosine similarity measure. Recommendations were then generated based on an aggregating preferences from from users deemed similar. This aggregation was weighted by their respective similarity scores, ensuring that more similar users had a greater influence on the recommendations.
Content-based approach
Within CB method, Lu 21 said that the KMeans clustering algorithm is predominantly employed to segment job roles into discrete clusters based on shared characteristics, such as skills and qualifications. The current study utilized multiple criteria for clustering, including skill relevance and job title similarities, resulting in informative and valuable clusters that accurately mirror the real-world grouping of job roles (Figure 3).
Job Positions Clustering (Source: Authors)
CB is a commonly employed technique that enables personalized recommendations to users. This technique involves the computation of similarity between an item and a user based on the item's features (1). Suriati et al.22 stated an item matrix A with element a showing the relationship between item i and feature j. Further, a rating matrix R with element r is also required, denoting the rating assigned by user u to item i. Suriati et al.22 stated that the fundamental objective behind this approach is to construct a user profile matrix B with element b signifying the relationship between user u and feature j. This can be accomplished by multiplying the rating matrix and the item matrix, as demonstrated in equation (2).
The two types of vectors, including item profile a and user profile b are used as indices to construct the cosine similarity, indicating the user's and item's similarity level. The score range, between -1 and 1, reflects the proximity between the vectors, with a score close to 1 indicating a high likelihood of match. Equation (3) is used to predict user ratings for items, with x representing the highest achievable rating within the system. This equation is based on the similarity score between the user and item vectors and allows us to predict users' preferences and provide recommendations accordingly.
Model Evaluation Metrics
To assess this RS's performance, a suite of evaluation metrics including Precision, Recall and F1-core were employed. These metrics provide a comprehensive understanding of the system's accuracy and effectiveness.
Precision
Sun et al.23 defined Precision in the context of this RS as the ratio of the True Positives (i.e., correctly recommended items) to the total number of items that the system classified as positive, which encompassed both True Positives (TP) and False Positives (FP). Mathematically, Precision was expressed as:
Where TP denoted True Positives and FP denoted FP. A higher Precision score indicated the system’s effectiveness in ensuring that the recommended learning paths were relevant to the user’s needs and preferences.
Recall
On the other hand, Solanki et al.24 stated that recall measured the system’s capability to identify all relevant items. It was calculated as the ratio of the TP to the sum of TP and False Negatives (FN), represented by:
In this scenario, a high Recall score implied that the system was adept at capturing a comprehensive range of suitable job positions and courses for the user.
F1-score
Chen et al. 25 noted that F1-Score provides a balanced system performance view by harmonizing Precision and Recall. This metric was the harmonic mean of Precision and Recall and was formulated as:
The F1-Score was a pivotal metric, especially in scenarios where there was an imbalance in the dataset or unequal distribution of classes, as it ensures that the recommendations' relevance and completeness are accounted for.
While it is essential to recognize the limitations of User-Based CF, which relies on existing employee interactions, this approach is highly effective at capturing and analyzing user preference patterns 24, 26. On the other hand, the CB approach has been criticized for its narrow focus on specific characteristics of items, such as courses or job roles, that employees have previously interacted with or shown interest in. However, this approach is instrumental in aligning recommendations with specific content attributes. By evaluating these two methods, the research identifies the inherent potential of each model to contribute uniquely to the development of RS in the context of employment and educational alignment. Incorporating both approaches allows for a more robust recommendation system capable of addressing a diverse range of user needs and scenarios27, 28. For instance, a hybrid model can mitigate the cold start problem associated with CF by utilizing CB recommendations for new users or items until sufficient interaction data becomes available. Conversely, the potential over-specialization of CB can be balanced by CF's ability to introduce diversity and serendipity into the recommendation mix.
Experimental Result and Discussion
Experimental Result
The empirical evaluation of this paper encompassed two distinct models: CF and CB. The performance of each model was rigorously assessed across three key metrics: Precision, Recall, and F1-Score. The results presented herein offer a clear, objective view of the models' effectiveness without delving into speculative interpretations.
Model Evaluation Metrics (Source: Authors)
|
Collaborative Filtering |
Content-based Filtering | |
|
Precision |
1.00 |
0.90 |
|
Recall |
0.83 |
0.86 |
|
F1-Score |
0.91 |
0.84 |
Discussion
Results Analysis
Collaborative Filtering Model
The first model under scrutiny was the CF approach. The Precision metric for this model was recorded at a perfect score of 1.00 (Table 1), signifying an exemplary level of accuracy where every recommended item (i.e., learning paths) was deemed relevant. This high Precision indicates the model's robustness in filtering out non-relevant recommendations, ensuring that learners are only presented with the most pertinent learning paths.
However, the Recall score was slightly lower at 0.83 (Table 1), implying that while the model was highly accurate in its recommendations, it did not capture the entire spectrum of relevant items. Such a scenario might lead to missing out on pertinent learning paths that could be beneficial for the learner.
The F1-Score, which is the harmonic mean of Precision and Recall, stood at 0.91 (Table 1). This score is significant as it demonstrates a balanced trade-off between Precision and Recall, underscoring the overall effectiveness of the CF model in providing relevant and comprehensive learning path recommendations.
Content-Based Model
The second model, the CB approach, demonstrated a slightly lower Precision score of 0.90. This indicates a minor reduction in accuracy compared to the CF model. While most recommendations were relevant, only a small fraction may have been entirely pertinent to the learners' needs.
In terms of Recall, the CB model scored 0.86 (Table 1), marginally outperforming the CF model. This higher Recall suggests that the CB model was more effective in identifying a broader range of relevant learning paths, albeit with a slight compromise in Precision.
The F1-Score for the CB model was calculated at 0.84 (Table 1). Although slightly lower than the CF model, this score still reflects a robust performance, indicating that the CB model is a viable alternative, particularly in scenarios where a broader identification of relevant items is prioritized over precision.
Implications
For University
Duan et al.29 identified the integration of recommendation systems within higher education frameworks as a pivotal strategy for enhancing curriculum relevance and ensuring alignment with labor market demands. Management factors such as strategic planning, stakeholder engagement, and continuous curriculum assessment play critical roles in this integration process28, 29. Strategic planning involves the adoption of forward-looking models that facilitate early identification of students’ career goals and academic interests, allowing universities, specifically in the context of this study, the UEL, to tailor their programs to better meet both student aspirations and the evolving needs of the industry. Forsythe30 provided the insight that stakeholder engagement, involving collaboration with industry partners, educators, and students, is essential for effectively understanding and responding to market trends and educational expectations.
Furthermore, continuous curriculum assessment ensures that academic offerings remain dynamic and responsive to changes in the labor market, thereby maintaining the applicability and value of the skills and knowledge taught 30, 31. The adoption of such systems necessitates universities to remain vigilant and responsive to current industry trends to preserve the relevance of their courses. The significance of leveraging technology in education, as highlighted by Smith and Worsfold 31, is supported by empirical evidence. Studies have shown that TEL can improve student engagement, higher retention rates, and better learning outcomes 10. Furthermore, Alamri et al. [32, p.339] stated that PLP has been increased student satisfaction and academic achievement.
FIS can also utilize data analytics to monitor and analyze trends within both student performance and industry requirements. This approach supports the adjustment of courses that are theoretically sound and practically relevant. Alamri et al. [32, p.331] discussed the potential of learning technology models to support personalization within blended learning environments in higher education. Studies by Cubit33 confirmed that personalized learning environments increase student engagement and achievement, illustrating the positive impact of technology-enabled personalization. Similarly, research by Vallée et al. 34 suggested that students in online and blended learning settings often achieve better outcomes compared to traditional classroom settings, thanks to the adaptability offered by TEL. Further supporting this, Freitas et al. 35 also found that personalized e-learning systems contribute to higher retention rates in higher education by addressing individual learning preferences and sustaining student interest.
Moreover, implementing RS should be considered part of a broader institutional change towards a more learner-centered approach. This shift requires a re-evaluation of teaching methodologies, assessment practices, and the overall student experience. The challenges and solutions associated with AI-based personalized e-learning systems are outlined in a study that point to the necessity of aligning educational technologies with pedagogical strategies and learning outcomes 36.
For Students
Xiao et al. 37 argued that students stand to benefit immensely from personalized educational experiences facilitated by RS. Such systems enable students to make informed decisions regarding their educational and career trajectories, enhancing their ability to align their training programs and course selections with their long-term professional goals. This personalized approach not only aids in students' professional and personal development but also fosters a more engaging and relevant learning experience. As illustrated by Alamri et al. [32, p.345], the ability to tailor one's academic path directly contributes to improved learning outcomes and better preparation for the job market. Longitudinal studies, such as those by the Bill & Melinda Gates Foundation, reinforced the value of personalized learning, indicating improved standardized test scores among students and increased confidence in their college and career prospects. This confidence, rooted in personalized educational experiences, paves the way for long-term success in both educational and professional arenas.
Furthermore, early exposure to career exploration platforms can significantly impact high school students, enabling them to make more informed decisions about their future education and employment opportunities. For instance, platforms like Naviance or Career Cruising offer personalized assessments that match students' interests and strengths with potential careers, guiding them toward relevant educational programs38. By engaging with these platforms, students can explore various career options, understand the educational requirements for each role, and plan their high school courses accordingly. This informed decision-making process ensures that students are better prepared for post-secondary education and the workforce with confidence and clarity, aligning their academic pursuits with their career aspirations and the current job market demands39.
For Businesses
From an employment perspective, RS would revolutionize the recruitment process by facilitating the identification of graduates whose education and skill sets align with specific job requirements. This alignment not only enhances the efficiency of the recruitment process but also optimizes resource utilization. Companies benefit from a streamlined recruitment process that is more closely aligned with industry trends, ultimately improving the quality and speed of the hiring process. The integration of such systems signifies a shift towards more data-driven and customized educational experiences, underscoring the mutual benefits of aligning educational programs with real-world applications and market needs.
However, it is imperative to critically examine their role in perpetuating or mitigating biases during the hiring process. Studies such as those by Gianfrancesco et al.40 revealed the inherent risk of these systems reinforcing existing societal and organizational biases, particularly when algorithms are trained on historical data that may reflect prejudiced hiring practices. This requires the need for deploying bias correction mechanisms and ensuring that recommendation systems are regularly audited for fairness.
This research makes significant theoretical and practical contributions to personalized learning and RS in IS. The study advances the understanding of how CF and CB approach can be tailored and integrated within the context of PLP. It also provides actionable insights for educators and developers on implementing these RSs to enhance educational content and career development tools. The research has the potential to pave the way for future studies on hybrid recommendation methodology, which suggests a new direction for combining different approaches to improve the personalization and effectiveness of learning paths in IS and other fields.
Conclusion and Future work
Conclusion
This project evaluated the accuracy and performance of the learning path RS using User-based CF and CB techniques separately. The research's findings confirmed the initial hypothesis that CF and CB models would each exhibit distinct strengths in PLP. The former model achieved an absolute Precision rate of 100%, while the latter excelled in Recall, identifying 85% of relevant learning paths. These insights extend beyond IS, suggesting potential applications in diverse educational fields, from digital marketing to healthcare training. The capability of CF and CB model to adapt to changing user preferences and the dynamic nature of IS sector content underscores their profound utility in real-world applications, ensuring that learning recommendations remain relevant and personalized, crucial for IS students seeking to stay abreast of technological advancements and emerging trends.
Limitations
When used independently, the performance of the proposed models has some specific limitations indicated by the application. For CF, essential barriers like data sparsity may reduce its ability to suggest new or uncommon learning paths. This obstacle arises because CF relies heavily on existing user interactions, making it difficult to suggest items with few or no ratings41. Conversely, the CB model, while effective in matching specific content attributes, may overlook the broader preferences and behavioral patterns of users, potentially limiting its ability to meet the diverse needs of learners in the IS.
Future Development
Future efforts will focus on developing a hybrid model, combining the behavioral analysis strength of CF with the precise content matching of the CB technique. This hybrid approach aims to mitigate the drawbacks of both models by integrating their strengths and proposing a more accurate and comprehensive PLP RS42. This approach directly addresses the research objective of evaluating the efficacy of different RS models in enhancing personalized educational experiences, aligning more closely with the progressed needs of IT education and career development. In addition, the current evaluation metrics, namely Precision, Recall, and F1-Score, focus primarily on the relevance and utility of the recommendation models. Moving forward, to better evaluate the hybrid model and provide a more nuanced understanding of its efficacy, it is crucial to incorporate metrics such as Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Normalized Discounted Cumulative Gain (NDCG). This metrics expansion will supplement the current evaluation framework, providing a deeper understanding of this research’s findings and the practical application of RS in education settings.
ACKNOWLEDGEMENTS
This research is funded by University of Economics and Law, Vietnam National University Ho Chi Minh City / VNU-HCM
LIST OF ABBREVIATIONS
RS: Recommender System
IS: Information Systems
CF: Collaborative Filtering
CB: Content-Based
FIS: Faculty of Information Systems
UEL: University of Economics and Law
CBRS: Content-Based Recommender Systems
TEL: Technology-Enhanced Learning
PLP: Personalized Learning Path
IT: Information Technology
KSA: Knowledge - Skill - Attitude
NLP: Natural Language Processing
TP: True Positives
FP: False Positives
FN: False Negatives
COMPETING INTERESTS
The author declares that there are no conflicts of interest in the publication of this article.
AUTHORS’ CONTRIBUTION
Tran Duong Thanh Phong and Ho Trung Thanh: Conceptualized and designed the study, Wrote the original manuscript, and Reviewed and Edited the article.
Vu Bao Khang and Ho Trung Thanh: Conducted the data preparation, Developed and implemented the algorithm, Conducted data analysis, Wrote the Experimental results.
Doan Nhat Minh: Conducted the data preparation, performed the analytical calculations, and Wrote the Discussion & Conclusion.
Dang Truc Quynh and Ho Trung Thanh: Conducted the Literature Review & Introduction, Contributed to the visualization of the study, and Reviewed the article.
Dang Viet Quang and Ho Trung Thanh: Assisted with the data preparation, Contributed to the experimental design, Wrote the Methodology, and Reviewed the article.