The deceptive simplicity of the brier skill score

Judith Bosboom*, Ad Reniers

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeChapterScientific

3 Citations (Scopus)

Abstract

The quality of morphodynamic predictions is often indicated by a skill score that weighs the mean-squared error of the prediction by that of the initial bed as the reference prediction. As simple as this Brier skill score (BSS) or meansquared- error skill score (MSESS) may seem, it is not well understood and, hence, sometimes misinterpreted. This chapter aims at improving the understanding of the MSESS. We review existing MSESS formulations and classifications, with and without an account of the measurement error. Using simple examples, we illuminate which aspects of prediction quality the MSESS actually measures. It is shown that the MSESS tends to favor model results that underestimate the variance of cumulative bed changes. We further demonstrate that the normalization by the observed cumulative change, which follows from the choice of the initial bed as the reference, is not effective in creating a level playing field over a wide range of prediction situations (trend, episodic event, different seasons). Also, it is shown that the combined presence of larger, persistent scales and smaller, intermittent scales in the cumulative bed changes may lead to an apparent increase of skill with time, although the prediction of neither of these scales becomes more skilful with time. Finally, in order to obtain a balanced appreciation of model performance, the use and development of a more extensive suite of validation measures is advocated.

Original languageEnglish
Title of host publicationHandbook of Coastal and Ocean Engineering
Subtitle of host publicationExpanded Edition
PublisherWorld Scientific Publishing
Pages1639-1663
Number of pages25
Volume2-2
ISBN (Electronic)9789813204027
ISBN (Print)9789813204010
DOIs
Publication statusPublished - 21 Dec 2017

Fingerprint

Dive into the research topics of 'The deceptive simplicity of the brier skill score'. Together they form a unique fingerprint.

Cite this