Formal methods: no cure for faulty reasoning

Abstract

Owing to the benefits commonly associated with their use and links with scientific culture, formal methods have become closely identified with the design of safety-critical systems. But, despite the mathematical nature of the logic systems underlying most formal notations, many aspects of formal methods are much less predictable than one might realise. Specifically, it is suggested that the ways in which people interpret and reason about formal descriptions can lead to similar kinds of errors and biases as those exhibited during previous cognitive studies of logical statements in natural language. This paper reports a series of preliminary experiments aimed at testing this hypothesis and several related issues. Early results suggest that, in reality, people frequently depart from fundamental principles of mathematical logic when reasoning about formal specifications, and are content to rely upon probablistic, heuristic methods. Furthermore, they suggest that manipulating such factors as the degrees of thematic and believable content in formal specifications can lead to significant reasoning performance enhancement or degradation. So, although faulty reasoning cannot be cured by formalisation alone, it would appear that the human potential for error can be reduced by avoiding certain expressions and choosing alternative, equivalent forms.