Machine-Crowd-Expert Model for Increasing User Engagement and Annotation Quality

Mendez, A.E.M., Cartwright, M., Bello, J.P. Machine-Crowd-Expert Model for Increasing User Engagement and Annotation Quality. In Proceedings of ACM CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2019.

Abstract

Crowdsourcing and active learning have been combined in the past with the goal of reducing annotation costs. However, two issues arise with using AL and crowdsourcing: quality of the labels and user engagement. In this work, we propose a novel machine _ crowd _ expert loop model where the forward connections of the loop aim to improve the quality of the labels and the backward connections aim to increase user engagement. In addition, we propose a research agenda for evaluating the model.