Competition-based rating system for medical website credibility

In this paper, we propose a new approach to the aggregation of monadic ratings (5-step scale) done by crowdsourcing users for the evaluation of medical websites. We compare them pairwise with other evaluations done by the same users for other websites (whether they are higher or lower), and we will use an Elo rating algorithm to calculate website “credibility” values. Results show that this method of crowdsourcing evaluation is highly correlated with expert evaluations. As proposed, a competition-based model uses a 5-step scale as ordinal and only compares which website is rated higher or lower by the same user. This approach can solve many problems associated with a 5-point scale, such as different understanding by users, user bias, and distribution skewness that can be clearly observed in results.