|
Hokkaido University Collection of Scholarly and Academic Papers >
Graduate School of Information Science and Technology / Faculty of Information Science and Technology >
Peer-reviewed Journal Articles, etc >
Separate or joint? Estimation of multiple labels from crowdsourced annotations
| Title: | Separate or joint? Estimation of multiple labels from crowdsourced annotations |
| Authors: | Duan, Lei Browse this author | | Oyama, Satoshi Browse this author →KAKEN DB | | Sato, Haruhiko Browse this author | | Kurihara, Masahito Browse this author →KAKEN DB |
| Keywords: | Multi-label estimation | | Crowdsourced annotation | | Label dependency | | Quality control | | Human computation |
| Issue Date: | 1-Oct-2014 |
| Publisher: | Elsevier |
| Journal Title: | Expert Systems with Applications |
| Volume: | 41 |
| Issue: | 13 |
| Start Page: | 5723 |
| End Page: | 5732 |
| Publisher DOI: | 10.1016/j.eswa.2014.03.048 |
| Abstract: | Artificial intelligence techniques aimed at more naturally simulating human comprehension fit the paradigm of multi-label classification. Generally, an enormous amount of high-quality multi-label data is needed to form a multi-label classifier. The creation of such datasets is usually expensive and time-consuming. A lower cost way to obtain multi-label datasets for use with such comprehension simulation techniques is to use noisy crowdsourced annotations. We propose incorporating label dependency into the label-generation process to estimate the multiple true labels for each instance given crowdsourced multi-label annotations. Three statistical quality control models based on the work of Dawid and Skene are proposed. The label-dependent DS (D-DS) model simply incorporates dependency relationships among all labels. The label pairwise DS (P-DS) model groups labels into pairs to prevent interference from uncorrelated labels. The Bayesian network label-dependent DS (ND-DS) model compactly represents label dependency using conditional independence properties to overcome the data sparsity problem. Results of two experiments, "affect annotation for lines in story" and "intention annotation for tweets", show that (I) the ND-DS model most effectively handles the multi-label estimation problem with annotations provided by only about five workers per instance and that (2) the P-DS model is best if there are pairwise comparison relationships among the labels. To sum up, flexibly using label dependency to obtain multi-label datasets is a promising way to reduce the cost of data collection for future applications with minimal degradation in the quality of the results. |
| Type: | article (author version) |
| URI: | http://hdl.handle.net/2115/57537 |
| Appears in Collections: | 情報科学院・情報科学研究院 (Graduate School of Information Science and Technology / Faculty of Information Science and Technology) > 雑誌発表論文等 (Peer-reviewed Journal Articles, etc)
|
Submitter: 栗原 正仁
|
