IEICE TRANSACTIONS on Information
Preventing Participation of Insincere Workers in Crowdsourcing by Using Pay-for-Performance Payments
Summary :
We propose a method for finding an appropriate setting of a pay-per-performance payment system to prevent participation of insincere workers in crowdsourcing. Crowdsourcing enables fast and low-cost accomplishment of tasks; however, insincere workers prevent the task requester from obtaining high-quality results. Instead of a fixed payment system, the pay-per-performance payment system is promising for excluding insincere workers. However, it is difficult to learn what settings are better, and a naive payment setting may cause unsatisfactory outcomes. To overcome these drawbacks, we propose a method for calculating the expected payments for sincere and insincere workers, and then clarifying the conditions in the payment setting in which sincere workers are willing to choose a task, while insincere workers are not willing to choose the task. We evaluated the proposed method by conducting several experiments on tweet labeling tasks in Amazon Mechanical Turk. The results suggest that the pay-per-performance system is useful for preventing participation of insincere workers.
- Publication
- IEICE TRANSACTIONS on Information Vol.E97-D No.9 pp.2415-2422
- Publication Date
- 2014/09/01
- Publicized
- Online ISSN
- 1745-1361
- DOI
- 10.1587/transinf.2013EDP7441
- Type of Manuscript
- PAPER
- Category
- Information Network
Authors
Shigeo MATSUBARA
Kyoto University
Meile WANG
Kyoto University
Keyword
Latest Issue
Copyrights notice of machine-translated contents
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Cite this
Copy
Shigeo MATSUBARA, Meile WANG, "Preventing Participation of Insincere Workers in Crowdsourcing by Using Pay-for-Performance Payments" in IEICE TRANSACTIONS on Information,
vol. E97-D, no. 9, pp. 2415-2422, September 2014, doi: 10.1587/transinf.2013EDP7441.
Abstract: We propose a method for finding an appropriate setting of a pay-per-performance payment system to prevent participation of insincere workers in crowdsourcing. Crowdsourcing enables fast and low-cost accomplishment of tasks; however, insincere workers prevent the task requester from obtaining high-quality results. Instead of a fixed payment system, the pay-per-performance payment system is promising for excluding insincere workers. However, it is difficult to learn what settings are better, and a naive payment setting may cause unsatisfactory outcomes. To overcome these drawbacks, we propose a method for calculating the expected payments for sincere and insincere workers, and then clarifying the conditions in the payment setting in which sincere workers are willing to choose a task, while insincere workers are not willing to choose the task. We evaluated the proposed method by conducting several experiments on tweet labeling tasks in Amazon Mechanical Turk. The results suggest that the pay-per-performance system is useful for preventing participation of insincere workers.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2013EDP7441/_p
Copy
@ARTICLE{e97-d_9_2415,
author={Shigeo MATSUBARA, Meile WANG, },
journal={IEICE TRANSACTIONS on Information},
title={Preventing Participation of Insincere Workers in Crowdsourcing by Using Pay-for-Performance Payments},
year={2014},
volume={E97-D},
number={9},
pages={2415-2422},
abstract={We propose a method for finding an appropriate setting of a pay-per-performance payment system to prevent participation of insincere workers in crowdsourcing. Crowdsourcing enables fast and low-cost accomplishment of tasks; however, insincere workers prevent the task requester from obtaining high-quality results. Instead of a fixed payment system, the pay-per-performance payment system is promising for excluding insincere workers. However, it is difficult to learn what settings are better, and a naive payment setting may cause unsatisfactory outcomes. To overcome these drawbacks, we propose a method for calculating the expected payments for sincere and insincere workers, and then clarifying the conditions in the payment setting in which sincere workers are willing to choose a task, while insincere workers are not willing to choose the task. We evaluated the proposed method by conducting several experiments on tweet labeling tasks in Amazon Mechanical Turk. The results suggest that the pay-per-performance system is useful for preventing participation of insincere workers.},
keywords={},
doi={10.1587/transinf.2013EDP7441},
ISSN={1745-1361},
month={September},}
Copy
TY - JOUR
TI - Preventing Participation of Insincere Workers in Crowdsourcing by Using Pay-for-Performance Payments
T2 - IEICE TRANSACTIONS on Information
SP - 2415
EP - 2422
AU - Shigeo MATSUBARA
AU - Meile WANG
PY - 2014
DO - 10.1587/transinf.2013EDP7441
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E97-D
IS - 9
JA - IEICE TRANSACTIONS on Information
Y1 - September 2014
AB - We propose a method for finding an appropriate setting of a pay-per-performance payment system to prevent participation of insincere workers in crowdsourcing. Crowdsourcing enables fast and low-cost accomplishment of tasks; however, insincere workers prevent the task requester from obtaining high-quality results. Instead of a fixed payment system, the pay-per-performance payment system is promising for excluding insincere workers. However, it is difficult to learn what settings are better, and a naive payment setting may cause unsatisfactory outcomes. To overcome these drawbacks, we propose a method for calculating the expected payments for sincere and insincere workers, and then clarifying the conditions in the payment setting in which sincere workers are willing to choose a task, while insincere workers are not willing to choose the task. We evaluated the proposed method by conducting several experiments on tweet labeling tasks in Amazon Mechanical Turk. The results suggest that the pay-per-performance system is useful for preventing participation of insincere workers.
ER -
English
