Interpreting palaeoecological data can be a opaque process, differing considerably between workers, and oftentimes scholars have some difficulty describing their own decision making process, or interpreting that of others, particularly in formal written formats like journal articles. This probably has a lot to do with the nature of multi-proxy palaeoecological investigations, where sources of information can be multiple, conflicting, incomplete, imprecise, and unreliable.
Often palaeoecological investigators don’t know exactly what information they will find in a palaeoecological archive before they analyse it, or what the quality of that information will be. This limits the use of statistical hypothesis test – for example, defining a hypothesis (and null hypothesis) to test for significance, although it has some limited application with quantitative data. Traditional hypothesis testing tends to focus on the most likely scenarios, rather than all of the proposed hypotheses. This got me thinking of other ways of testing hypotheses with palaeoecological data.
In many ways, palaeoecological data is a lot like intelligence, medical, or forensic data – information is derived from multiple, different, incomplete, unreliable sources, and can be interpreted in different ways. It is comprised of imperfect evidence preserved after some event or epoch, and it is up to the researcher(s) to compose different information sources into some coherent, plausible sequence of events, causal explanation and/or quantitative information about a past environment. This led me to take a look at methodical ways of testing hypotheses used in other fields.
For example, anyone who has seen the medical drama “House, M.D.” will be familiar with the fast-paced “differential diagnosis” sessions Gregory House (Hugh Laurie) holds with his team. The system is commonly taught in medical schools to assist medical practitioners to come to a diagnosis of a patient’s condition when the symptoms presented are similar. It also allows a medical practitioner to select an appropriate diagnostic if they are unable to discriminate between two or more diagnoses. The process can be broadly summarised as:
- Gather all information.
- List all possible causes.
- Prioritise the list by risk to the patient’s health.
- Working from the highest priority to the lowest, rule out each condition using the available information.
This simple model perhaps mirrors the approach informally adopted by many palaeoecological workers – collect data, hypothesise, rule out until settled on answers. This model fails to accommodate the possibility of competing hypotheses that cannot be adequately differentiated because of limitations in the information available. The intelligence analytical community has developed a way of reasoning and testing hypotheses called the “Analysis of Competing Hypotheses” (ACH). This approach can accommodate the various imperfections of the information available, and can indicate (qualitatively) the likelihood of a particular hypothesis being false. To summarise, the process goes something like this:
- Identify the possible hypotheses.
- List information and arguments (inc. assumptions and deductions) both for and against each hypothesis.
- Assess the relative “diagnosticity” of each piece of information.
- Prepare a matrix with hypotheses in columns, and all evidence and/or arguments in rows.
- Assess how consistent each piece of information or argument is with each hypothesis, attempting to refute each hypothesis.
- Reconsider the hypotheses, removing sources that don’t help discriminate, and identify further evidence required.
- Iterate steps 2-7 as required.
- Draw tentative conclusions about the relative likelihood of each hypothesis (rank them).
- Consider how sensitive your conclusion is the a few critical items of information, and the consequences thereof.
- Report conclusions, discussing all hypotheses.
ACH was introduced by Richard Heuer in “The Psychology of Intelligence Analysis” (CIA) to combat confirmation bias in the field of intelligence analysis, to facilitate multiple workers to address a common problem with multiple lines of evidence, and to create an audit trail for intelligence decisions.
It’s clear that with multiple lines of evidence, weighting, and the iterations, this could quickly become more complicated than just muddling through the data. This is perhaps why there is a growing market for consultants marketing their software and services in intelligence, forensics and criminal investigation. Fortunately both Richard Heuer’s treatise on the subject, alongside some powerful software to assist, are available gratis online.
I’d be interested to hear from anyone who’d like to try (or has tried) using ACH in their analysis of palaeoenvironmental data. I’ll happily configure a portable web-server if you’d like to try the software based version in a group meeting.