Metric-Dependent Annotation Saturation for Learning from Label Distributions

Apple researchers have demonstrated that the number of human annotators required to train a machine learning model effectively depends on the specific performance metric being targeted. By analyzing label distributions from the ChaosNLI dataset, the study shows that disagreement among annotators provides useful data that can improve model accuracy when captured appropriately.