Devilly & Spence, logical analyses and other stuff

Ricky:

I didn't mean to surprise you by citing Devilly & Spence. As I'm sure your aware, I've discussed the weaknesses of the study desing both here on this discussion group and in my Psychiatric Times piece, to which Sandra previously posted a link (for anyone who is interested). So, why would I cite this study as "as an example of CBT having a better retention rate than EMDR"? The simple answer is that I did not cite it as evidence CBT has a lower dropout rate than EMDR. In fact, I've never made such a claim. Rather, I was listing all evidence to bear on YOUR hypothesis that EMDR has less drop out the CBT.

Now, why would it make a difference as to which hyothesis I'm evaluating? Because, logically speaking evidence that disconfirms your hypothesis has far more weight in evaluating the validity of your hypothesis than evidence that is consistent with your hypothesis. Here is the reason. When you review evidence that is in support of your hypothesis, you are engaging in a sort of reasoning that follows the formula:

P1. If theory T is correct, then I should should be able to make observation O.
P2. I make observation O.
C. Therefore, theory T is true.

This form of reasoning is very common. It is so common, logicians have a special name for it: Affirming the consequent. It is also widely recognized as being formally invalid, which means that the conclusions (the therefore clause) do not necessarily follow from the preceeding propositions. In other words, both of the propositions could be true, but the conclusion could none-the-less be false. Consider the following example:

P1. If Elvis presly was assasinated, then he is dead.
P2. Elvis is dead.
C. Therefore he was assasinated.

Both propositions are true, but the conclusion is false. Elvis died of heart-attack.

Now, consider a different kind of reasoning, called modus tollens by logicians. It follows the form:

P1. If theory T is correct, then I should should be able to make observation O.
P2. I observer not-O.
C. Therefore, theory T is wrong.

This form of reasoning is formally valid, such that if the two propositions are true then the conclusion MUST be true. Let's apply it to my Elvis analogy:

P1. If Elvis presly was assasinated, then he is dead.
P2. Elvis is not dead.
C. Therefore he was not assasinated.

Please susspend reality for a moment and assume that the King is still alive. You can see that if both propositions are true, then the conclusion is true.

Now, your hypothesis is that EMDR has less drop out than exposure therapy. And you can site evidence in support of your hypothesis. However, you must also consider evidence contrary to your hypothesis. Devilly & Spence, although flawed, provides evidence that is directly contrary to your hypothesis. Thus, it is of importance to take into consideration. However, I would never use it as positive evidence for the hypothesis that CBT has lower dropout than EMDR. In fact, my own hypothesis is that the dropout rates don't differ. Unfortunately, I'll never be able to prove my hypothesis because one can't prove the null. Which is why I've never bothered to advance it.

Also, I'll remind you that the Van Etten & Taylor meta-analysis that drew conclusions about the relative efficacy of EMDR and CBT did not include a single study in which patients were randomly assigned to one treatment or another. Thus, the meta-analytic comparison between EMDR and CBT is based on comparisons of one group of EMDR studies with a different group of CBT studies. Yet, I've seen you cite it's conclusions without reservation. So, if randomization is your standard, then you need to stop citing van Etten and Taylor. If you consider the van Etten and Taylor results valid, then you cannot reject Devilly & Spence on grounds of non-randomization. A little consistency please.

Now, regarding your other criticisms of Devilly & Spence, I happen to not share your concerns about his having done EMDR poorly, but that's a more detailed discussion we can perhaps persue another day. Regarding his supposed poor outcome with EMDR, your characterization is oversimplified. Also, the simple fact is that you can't use treatment outcome to make valid inferences about treatment fidelity. Interestingly, I made these points in greater detail in a previous post which (11/24/02). Apparently you missed it. For you convenience, I reproduced it below. I particularly call your attention to the results of the interviewer measure of PTSD, generally considered to be the "gold statnard" in treatment outcome studies. Interestingly, the first point I made has to due with the logical fallicy of affirming the consequence, although I don't label it as such. Boy, the more things change, the more they stay the same.

Cahill · 11/24/02 at 7:23 PM ET
A few quick comments in response to Ricky's post.

First, you cannot make logical inferences about the quality of treatment from outcome. This involves the logical error of affirming the consequent. The logic of the argument is that (a) If treatment is done poorly, then outcome will be limited; (b) outcome was limited; therefore (c) treatment was done poorly. While it is possible that the reason for poor outcome was poor implementation, it is not the only possible reason, not the least of which is the possibility that the treatment just didn't work very well in the population under study.

Second, the results of the Devilly and Spence study are not all so bad. The Table below summarizes the within group effect sizes from the Devilly and Spence study for measures of self-reported anxiety (trait portion of the State-Trait Anxiety Inventory), self-reported depression (Beck Depression Inventory), self-reported PTSD symptom severity (the Civilian Mississippi Scale, the Impact of Event Scale, and the PTSD Symptom Scale - Self-Report), and interviewer rated PTSD symptom severity (PTSD-Interview). For anxiety, the ES in the Devilly & Spence study fell within the 90% Confidence Interval from the Van Etten and Taylor meta-analysis. For depression, the ES from Devilly and Spence fell below the 90% confidence interval from Van Etten and Taylor and similarly for self-reported PTSD symptoms. However, for interviewer rated PTSD severity, the effect size from Devilly & Spence was a whopping 2.37, which was higher than the 90% confidence interval from Van Etten and Taylor.

Self-reported anxiety
Devilly & Spence .79
Van Etten & Taylor .95 (.69-1.21)

Self-reported depression
Devilly & Spence .68
Van Etten & Taylor 1.05 (.81-1.29)

Self-reported PTSD Sx
Devilly & Spence .75, .74, .75
Van Etten & Taylor 1.21 (.99-1.49)

Interviewer rated PTSD Sx
Devilly & Spence 2.37
Van Etten & Taylor.69 (-.06-1.44)

Third, the arguments against treatment fidelity within the Devilly and Spence study ignores the fact that someone who was (a) Level II trained in EMDR and (b) not otherwise associated with the research (i.e., had no investment in the outcome) rated all available video tapes of EMDR sessions and provided a very high rating of treatment fidelity.

Replies:

Re:Devilly & Spence, logical analyses and other stuff, by Realist, 05/02/03

| Behavior OnLine Home Page | Disclaimer |

Copyright © 1996-2004 Behavior OnLine, Inc. All rights reserved.