James C. Cavendish, General Motors Research and Development Center
Computer simulations and math models are used in industry to design, understand
and control various physical systems and processes and to predict the performance
of these systems and processes under various operating conditions. The two main
approaches to establishing credible confidence statements about the predictive
capabilities of computer models are verification and validation. Verification
is the process of determining that a computer model (that is, the operational
computer program or "code") is a correct (i.e. bug-free) and accurate
implementation of a conceptual model of a physical system or process. Model
validation is the process of determining the degree to which output from a computerized
model accurately represents reality. In this talk we present a framework that
enables the implementation of the validation process.
This framework is being developed as part of a Research Agreement between the
National Institute of Statistical Sciences (NISS) and General Motors. Underlying
the proposed validation framework is a systematic six-step procedure based upon
a Bayesian statistical methodology. The Bayesian methodology is particularly
suited to treating the major issues associated with the validation process:
quantifying multiple sources of error and uncertainty in math models; combining
multiple sources of information; and updating validation assessments as new
information is acquired. Moreover, it allows inferential statements to be made
about predictive error associated with model predictions in untested situations
(that is, prediction beyond the validation comparisons between calculations
and experimental outcomes). The framework is described with the help of a test
bed model that provides context for each of the six steps in the proposed validation
process.
See related publication: M.J. Bayarri, J.O. Berger, D. Higdon, M. Kennedy, A. Kottas, R. Paulo, J. Sacks, J.A. Cafeo, J.C. Cavendish, C-H lin, J. Tu (2002), "A Framework for Validation of Computer Models", Technical Report 128, NISS www.niss.org/downloadabletechreports.html.