Standard

Early Experiences with Crowdsourcing Airway Annotations in Chest CT. / Cheplygina, Veronika; Perez-Rovira, Adria; Kuo, Wieying; Tiddens, Harm A.W.M.; Bruijne, M de.

Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Proceedings. ed. / G. Carneiro; D. Mateus; L. Peter; A. Bradley; J.M.R.S. Tavares; V. Belagiannis; J.P. Papa; J.C. Nascimento; M. Loog; Z. Lu; J.S. Cardoso; J. Cornebise. Cham : Springer, 2016. p. 209-2018 (Lecture Notes in Computer Science; Vol. 10008).

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Harvard

Cheplygina, V, Perez-Rovira, A, Kuo, W, Tiddens, HAWM & Bruijne, MD 2016, Early Experiences with Crowdsourcing Airway Annotations in Chest CT. in G Carneiro, D Mateus, L Peter, A Bradley, JMRS Tavares, V Belagiannis, JP Papa, JC Nascimento, M Loog, Z Lu, JS Cardoso & J Cornebise (eds), Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Proceedings. Lecture Notes in Computer Science, vol. 10008, Springer, Cham, pp. 209-2018, First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Athens, Greece, 21/10/16. https://doi.org/10.1007/978-3-319-46976-8_22

APA

Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H. A. W. M., & Bruijne, M. D. (2016). Early Experiences with Crowdsourcing Airway Annotations in Chest CT. In G. Carneiro, D. Mateus, L. Peter, A. Bradley, J. M. R. S. Tavares, V. Belagiannis, J. P. Papa, J. C. Nascimento, M. Loog, Z. Lu, J. S. Cardoso, & J. Cornebise (Eds.), Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Proceedings (pp. 209-2018). (Lecture Notes in Computer Science; Vol. 10008). Springer. https://doi.org/10.1007/978-3-319-46976-8_22

Vancouver

Cheplygina V, Perez-Rovira A, Kuo W, Tiddens HAWM, Bruijne MD. Early Experiences with Crowdsourcing Airway Annotations in Chest CT. In Carneiro G, Mateus D, Peter L, Bradley A, Tavares JMRS, Belagiannis V, Papa JP, Nascimento JC, Loog M, Lu Z, Cardoso JS, Cornebise J, editors, Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Proceedings. Cham: Springer. 2016. p. 209-2018. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-319-46976-8_22

Author

Cheplygina, Veronika ; Perez-Rovira, Adria ; Kuo, Wieying ; Tiddens, Harm A.W.M. ; Bruijne, M de. / Early Experiences with Crowdsourcing Airway Annotations in Chest CT. Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Proceedings. editor / G. Carneiro ; D. Mateus ; L. Peter ; A. Bradley ; J.M.R.S. Tavares ; V. Belagiannis ; J.P. Papa ; J.C. Nascimento ; M. Loog ; Z. Lu ; J.S. Cardoso ; J. Cornebise. Cham : Springer, 2016. pp. 209-2018 (Lecture Notes in Computer Science).

BibTeX

@inproceedings{fad9dcf1ab8d4537824c0c9c0bda069c,
title = "Early Experiences with Crowdsourcing Airway Annotations in Chest CT",
abstract = "Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.",
author = "Veronika Cheplygina and Adria Perez-Rovira and Wieying Kuo and Tiddens, {Harm A.W.M.} and Bruijne, {M de}",
year = "2016",
doi = "10.1007/978-3-319-46976-8_22",
language = "English",
isbn = "978-3-319-46975-1",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
pages = "209--2018",
editor = "G. Carneiro and D. Mateus and L. Peter and A. Bradley and J.M.R.S. Tavares and V. Belagiannis and J.P. Papa and J.C. Nascimento and M. Loog and Z. Lu and J.S. Cardoso and J. Cornebise",
booktitle = "Deep Learning and Data Labeling for Medical Applications",
note = "First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016 : Held in Conjunction with MICCAI 2016 ; Conference date: 21-10-2016 Through 21-10-2016",

}

RIS

TY - GEN

T1 - Early Experiences with Crowdsourcing Airway Annotations in Chest CT

AU - Cheplygina, Veronika

AU - Perez-Rovira, Adria

AU - Kuo, Wieying

AU - Tiddens, Harm A.W.M.

AU - Bruijne, M de

PY - 2016

Y1 - 2016

N2 - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

AB - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

U2 - 10.1007/978-3-319-46976-8_22

DO - 10.1007/978-3-319-46976-8_22

M3 - Conference contribution

SN - 978-3-319-46975-1

T3 - Lecture Notes in Computer Science

SP - 209

EP - 2018

BT - Deep Learning and Data Labeling for Medical Applications

A2 - Carneiro, G.

A2 - Mateus, D.

A2 - Peter, L.

A2 - Bradley, A.

A2 - Tavares, J.M.R.S.

A2 - Belagiannis, V.

A2 - Papa, J.P.

A2 - Nascimento, J.C.

A2 - Loog, M.

A2 - Lu, Z.

A2 - Cardoso, J.S.

A2 - Cornebise, J.

PB - Springer

CY - Cham

T2 - First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016

Y2 - 21 October 2016 through 21 October 2016

ER -

ID: 11606755