home    about    browse    search    latest    help 
Login | Create Account

Accounting for uncertainty in multi-criteria sustainability assessments at the farm level: Improving the robustness of the SMART-Farm Tool

Schader, Christian; Curran, Michael; Heidenreich, Anja; Landert, Jan; Blockeel, Johan; Baumgart, Lukas; Ssebunya, Brian; Moakes, Simon; Marton, Silvia; Lazzarini, Gianna; Niggli, Urs and Stolze, Matthias (2019) Accounting for uncertainty in multi-criteria sustainability assessments at the farm level: Improving the robustness of the SMART-Farm Tool. Ecological Indicators, 106, p. 105503.

[thumbnail of schader-etal-2019-EcologicalIndicators-Vol106-105503_SMARTUncertainty.pdf] PDF - English
Limited to [Depositor and staff only]


Document available online at: https://www.sciencedirect.com/science/article/pii/S1470160X19304881


Many farm sustainability assessments use multi-criteria methods for aggregating indicators based on performance scores and importance weights. One of these is the SMART-Farm Tool, which measures the degree of goal achievement of farms across 327 indicators nested within 4 sustainability dimensions, 21 themes and 58 sub-themes of the FAO-SAFA Guidelines (Sustainability Assessment of Food and Agriculture). This study aims to improve the empirical foundation of the SMART-Farm Tool by (i) evaluating the uncertainty of indicator weights obtained via expert opinion and (ii) integrating this uncertainty into SMART assessment results. An adapted Delphi process was implemented, involving a group of 67 experts from 21 countries divided into thematic and regional sub-groups. Experts within each sub-group rated the importance of indicators of all 58 SAFA sub-themes, and self-assessed their own competence. Contrary to expectations, the uncertainty in expert opinions was relatively high for environmental indicator weights (although lowest for animal welfare), being comparable to social and governance indicators. Considerable uncertainty remained in indicator weights, even after two rounds of discussion and exchange of views. This is attributed to regional variation and inherent system complexity (i.e. experts having legitimate but diverging viewpoints based on contradictory evidence) rather than scientific ignorance (i.e. a lack of research evidence). Nevertheless, it is expected that the levels of uncertainty could be reduced by limiting the number of indicators to be evaluated and thus by allowing for more in-depth discussions among the experts. Monte Carlo Simulations were used to translate residual indicator weight uncertainty into SMART assessment results at the farm level for four example farms from both developed and developing countries. The resulting comparisons revealed several cases where substantial apparent differences between farms in sustainability scores for a specific sub-theme (up to 23%) were not statistically significant, while in other cases differences of 5% were significant. This emphasizes the general importance of considering uncertainty in multi-criteria assessment tools, with clear implications real-world applications, such as product certification, labelling and marketing. Finally, this study provides important methodological suggestions for implementing expert-based assessments in multi-criteria assessments efficiently.

EPrint Type:Journal paper
Keywords:SMART, sustainability, Multi-criteria assessment, Agriculture, Expert opinion, Delphi, Nominal Group Technique, Monte Carlo Simulations
Subjects: Food systems > Policy environments and social economy
Knowledge management > Education, extension and communication
Research affiliation: Switzerland > FiBL - Research Institute of Organic Agriculture Switzerland > Society > Agri-food policy
Switzerland > FiBL - Research Institute of Organic Agriculture Switzerland > Sustainability > Sustainability assessment
Related Links:https://www.fibl.org/en/projectdatabase/projectitem/project/738.html, https://www.fibl.org/en/themes/smart-en.html
Deposited By: Schader, Dr. Christian
ID Code:36325
Deposited On:07 Aug 2019 09:15
Last Modified:16 Feb 2022 15:41
Document Language:English
Refereed:Peer-reviewed and accepted

Repository Staff Only: item control page


Downloads per month over past year

View more statistics