The Development and Evaluation of E-Diagnostic Application Software

This paper describes the development of a new system that aims to identify the reason(s) behind learners‘ mistakes in solving a linear equation. The system works by studying the learner‘s steps during solving process, and checking whether there is any missing knowledge required for a correct solution. The system then provides, if appropriate, a correct solution path to learners, before retesting them.

In the first prototype, the system proved successful in identifying missing knowledge, but its UI (User Interface) was too complex and difficult to use. This issue was resolved, however, with the help of models and algorithms, which facilitated the development of a more user- friendly UI. Resulting tests showed that, in many cases, learners were successful in passing the test after following the remedial path suggested by the system. The evaluation, by the system, of the learners‘ solution process was confirmed by the (human) teacher in the majority of the cases, and learners reported finding the system helpful and easy to use.

The current study conducted five experiments. First experiment was usability evaluation to evaluate the concept of this study in real-world tests. The experiment found a significant improvement in the user interface (4.69 out of 5) as a result of using multi models which improve the UI from the first prototype. This experiment helped in understanding learner preferences. The second experiment was conducted to confirm the readiness of the system after fixing all the bugs and programming errors which were discovered during the usability evaluation experiment. In this experiment, students had to make specific errors intentionally to retrieve the system's action to those types of errors. The third and the main experiment was about the functionality evaluation which evaluated the concept of the research. With an average score of 3.32 out of 5, this experiment showed the system‘s efficiency in helping learners to determine their missing skills and find the right domain to learn. However, with polarity in the result and by reviewing the repetition table, this repetition is mainly in the 5 out of 5 score. This is because most students gave high scores in all of the questions. Learners become familiar with using the system, gradually as they use the system and most of them were excited to learn from the system about their mistakes. The teachers' opinion was positive to apply such a system for the other subjects. The fourth experiment was about the teachers' evaluation of the prototype, which was quite important to the research, and its different sequences. The results of this experiment shows the quantitative data about the usability and the functionality of the system with an average score of 3.9 out of 5 indicating improvement of the system. Qualitative data with an average score of 4.3 out of 5 indicates high level of satisfaction of the teachers about such a system. The data shows the high demand for using Sequence 1. However, there are some opinions about using Sequence 2 and some concerns about the feasibility of designing such a system in Sequence 1, which needs a lot of effort, whereas Sequence 3 can be implemented faster and easier. The fifth experiment showed the comparison of the time consumed between Sequence 1 and Sequence 3. The result of this experiment shows an average of 16:57 for Sequence 1, while the average of Sequence 3 was 28:46. This result supports the importance of Sequence 1, even if its cost is higher than Sequence 3.

The results of the study were positive enough to suggest that the concept could be applied to other domains involving a learning or teaching process. Such a system may be helpful, for example, in evaluating teachers‘ performances.