The value of replicationg the data analysis of an empirical evaluation

Abstract

The aim of this research was to determine whether the results of an empirical evaluation could be confirmed using a different evaluation method. In this investigation the Qualitative Weight and Sum method used by the researchers Graf and List to evaluate several free and open source e-learning software platforms, were studied and replicated with a different method.
The Analytical Hierarchy Process was used as an alternative to the Qualitative Weight and Sum method. The same input data used by the researchers Graf and List were used to replicate the evaluation.
It was found that the ranking of the e-learning platforms, when using the Qualitative Weight and Sum method, differs slightly from the outcome with the Analytical Hierarchy Process method. The Analytical Hierarchy Process method has an advantage over the Qualitative Weight and Sum method in that it determines the consistency of the evaluation judgments of evaluators. Both methods are useful but it is easier to rank software using the Analytical Hierarchy Process. It was found that replication of evaluation, which was done to validate the process and results, and also to determine the consistency of evaluators was useful.