The Limits of Psychology Research

Hans Eysenck, Again

In an earlier post, we discussed psychologist Hans Eysenck. To recap, in 1952, Eysenck published a review arguing that people with neurotic problems recovered at about the same rate whether or not they had psychotherapy (Eysenck, 1952). It took 25 years for research to offer a convincing answer (Cuijpers et al., 2018). In 1977, Mary Lee Smith and Gene Glass pooled hundreds of controlled studies and found that the average treated person ended up better off than about 75% of untreated people (Smith & Glass, 1977). Eysenck did not accept the result. He called the analysis “an exercise in mega-silliness,” objecting that it blended very different patients, therapies, and measures, and that many of the included studies were poorly designed (Eysenck, 1978).

Eysenck’s later record gives a reason to check his claims instead of taking them on his reputation. In 2019, King’s College London conducted an inquiry into his joint work with Ronald Grossarth-Maticek on personality and disease and judged 26 of their papers “unsafe,” meaning they could not be relied on as scientifically sound. The inquiry cited doubts about how the datasets were collected and results with no parallel in medicine (King’s College London, 2019). A flawed researcher can still raise a fair objection, though. The “apples and oranges” and “garbage in, garbage out” problems with pooling very different studies are now standard concerns in meta-analysis. A reanalysis of the depression literature found that unadjusted meta-analyses overestimate the effects of therapy, and it concluded that Smith and Glass probably overestimated them too (Cuijpers et al., 2018), a conclusion other researchers dispute, as discussed below.

A correction: in “Does therapy work?” I described the King’s College London inquiry as finding questionable research practices and data manipulation in Eysenck’s work, and I placed it next to his criticism of psychotherapy. The inquiry covered his joint papers on personality and disease, and it concluded that 26 papers were unsafe. It did not examine his psychotherapy review. I have corrected that post.

The Replication Crisis and the File Drawer

Psychology’s “replication crisis” is the name for a run of discoveries in the early 2010s showing that many published findings could not be repeated. A paper claiming evidence for extrasensory perception passed peer review at a major journal, a well-known social psychologist was found to have fabricated data, and researchers showed how easily ordinary analysis choices can produce significant results from noise (Pashler & Wagenmakers, 2012).

In 2015, a team called the Open Science Collaboration tried to repeat 100 published psychology studies (Open Science Collaboration, 2015). Ninety-seven percent of the originals had reported statistically significant results. Thirty-six percent of the repeats did, and the repeated effects averaged about half the original size. Many people took this as the crisis in numbers.

The number was disputed. A group of psychologists argued that the paper contained statistical errors and that its data were consistent with high reproducibility (Gilbert et al., 2016). The original authors replied that both optimistic and pessimistic readings of their data were possible, and that neither was yet warranted (Anderson et al., 2016).

Whether “crisis” is the right word is open to debate, and the 36% figure should not be read as an estimate for therapy research, since the 100 studies came from three journals. What carries over is a pattern that has been measured in therapy research directly. Not every trial gets published, and for depression trials funded by the US National Institutes of Health, adding the unpublished trials lowered the average effect (Driessen et al., 2015).

Problems Specific to Therapy Research

Therapy trials run into problems that drug trials mostly avoid. In a medication trial, patients and doctors can be kept from knowing who gets the real pill. In a therapy trial, patients and therapists cannot be kept unaware of who is in treatment, and people who know they are getting help may improve partly because they expect to (Cuijpers et al., 2018).

The comparison group matters too. Many trials compare therapy with a waiting list, and the waiting list seems to make therapy look better than it would against other comparisons, a point researchers dispute (see below). A network meta-analysis of 49 trials of cognitive behavioral therapy for depression found larger effects when the control group was a waiting list than when it was no treatment or a psychological placebo, which suggests that waiting itself may leave people worse off (Furukawa et al., 2014). In the depression trials Cuijpers and colleagues analyzed, the average effect was g = 0.89 against a waiting list and g = 0.61 against usual care (Cuijpers et al., 2018).

Researchers also tend to believe in the therapy they study, and that belief tracks their results. A review of meta-analyses found a moderate and consistent link between a researcher’s allegiance to a therapy and how well that therapy performed (r = .26; Munder et al., 2013). Another analysis found the link held even when the treatments being compared were thought to work about equally well, and allegiance explained 12% of the variation in outcomes (Munder et al., 2012). Others caution that the link may reflect a confound and does not prove a cause (Berman & Reich, 2010). Even the allegiance research shows the pattern: authors who believed in allegiance bias found larger effects of it (Munder et al., 2013). The EMDR post ran into the same thing.

Finally, many trials are too small to tell which therapy works best for which person. A review of 41 depression trials that tested whether particular patient characteristics predict which therapy works best estimated that, at the current pace, it would take another 326 years to gather enough evidence to detect a medium-sized effect (Cuijpers et al., 2016).

Checking My Own Sources

In “Does therapy work?” I made three claims. Here is how each holds up.

Therapy works. This survives, but how big the benefit is depends on whom you ask. The 1977 meta-analysis I cited corresponds to an effect size of 0.68, meaning the average treated person did better than about 75% of untreated people (Smith & Glass, 1977; Cuijpers et al., 2018). Cuijpers and colleagues reanalyzed the depression trials and found a similar unadjusted effect of 0.70. After limiting the analysis to Western-country trials, removing waiting-list controls and trials at risk of bias, and adjusting for publication bias, it fell to 0.31(Cuijpers et al., 2018), which by the same conversion is about 62% (my calculation). Only 23% of the studies had low risk of bias. Another group, including Bruce Wampold, answered that waiting-list trials are the right benchmark for Eysenck’s question about natural recovery, and that using them gives an effect near 0.70, with smaller but real effects against usual care (0.31) and other controls (0.43) (Munder et al., 2018). Cuijpers’s group replied that waiting lists may suppress natural recovery and inflate the numbers (Cuijpers et al., 2019). Cuijpers’s group concludes that therapy for depression probably works while noting that it cannot rule out small effects, and Munder’s group concludes that it is effective. They disagree about how large the effect is, and the disagreement turns on whether a waiting list is a fair comparison.

The type of therapy doesn’t matter. For depression, the evidence supports small differences at most. A meta-analysis of 53 comparison studies found no meaningful differences except that interpersonal therapy did slightly better and supportive counseling slightly worse (Cuijpers et al., 2008), and a network meta-analysis agreed (Barth et al., 2013). The picture is less clean when CBT is compared directly with psychodynamic therapy, including for depression. A 2010 meta-analysis found CBT outperformed psychodynamic therapy for anxiety and depression (Tolin, 2010), and a reanalysis found no CBT advantage on outcomes beyond disorder-specific symptoms (Baardseth et al., 2013). I leaned on Bruce Wampold’s work for this claim. He is the leading advocate of the common-factors view and a co-author of that reanalysis, so the same caution I applied to Gerald Rosen in the EMDR post applies here.

The relationship predicts outcome. A meta-analysis of 295 studies and over 30,000 patients found a consistent link between alliance and outcome (r = .28) across treatment approaches (Flückiger et al., 2018). An association that size is reliable and leaves most of the variation in outcomes unexplained. Direction is the open question. An analysis of the first seven sessions found that alliance and symptoms influence each other (Flückiger et al., 2020), and one review argues that part of the alliance reflects what clients bring to therapy, with only the part that changes during treatment helping in its own right (Zilcha-Mano, 2017). Wampold co-authored both Flückiger papers.

What Holds Up

Everything above could be read as a reason to dismiss psychology research. The evidence supports a narrower conclusion: some claims have survived scrutiny from researchers who disagree with each other, and others have not.

Therapy for depression belongs in the first group. The researchers who think the early numbers were inflated and the researchers who think they were roughly right both conclude that it works. Cuijpers’s group concludes that it probably works, and Munder’s group concludes that it does. They disagree about how much (Cuijpers et al., 2018; Munder et al., 2018).

The same research shows how to read new claims. Trials at higher risk of bias reported bigger effects, averaging 1.11 in the highest-risk studies (only 14 studies) and 0.46 in the lowest-risk ones (Cuijpers et al., 2018), and researchers who favor a therapy tend to find bigger effects for it (Munder et al., 2013). In the replication project, results backed by stronger original evidence were more likely to hold up (Open Science Collaboration, 2015). A single exciting study, or an effect with no parallel in medicine, deserves the caution that the Grossarth-Maticek results called for.

For someone choosing a therapist, the differences between established therapies for depression are small (Cuijpers et al., 2008), though researchers still disagree about whether CBT has an edge over psychodynamic therapy (Tolin, 2010; Baardseth et al., 2013). The alliance reliably tracks outcome (Flückiger et al., 2018). For trauma, guidelines are more specific. The American Psychological Association’s 2025 guideline lists cognitive processing therapy, prolonged exposure, and trauma-focused CBT as first-line treatments and EMDR as second-line (American Psychological Association, 2025), a ranking EMDR researchers dispute (Lee et al., 2026; see also the EMDR post). A therapist you can be honest with, who practices an approach with good evidence for your problem, is a sound place to start.

References

American Psychological Association. (2025). Clinical practice guideline for the treatment of posttraumatic stress disorder (PTSD) in adults. https://www.apa.org/ptsd-guideline

Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., Della Penna, N., Estel, V., Fedor, A., Fitneva, S. A., Frank, M. C., Grange, J. A., Hartshorne, J. K., Hasselman, F., Henninger, F., . . . Zuni, K. (2016). Response to comment on “Estimating the reproducibility of psychological science.” Science, 351(6277), 1037. https://doi.org/10.1126/science.aad9163 [Author list and order were taken from a repository listing and not checked against the journal page.]

Baardseth, T. P., Goldberg, S. B., Pace, B. T., Wislocki, A. P., Frost, N. D., Siddiqui, J. R., Lindemann, A. M., Kivlighan, D. M., Laska, K. M., Del Re, A. C., Minami, T., & Wampold, B. E. (2013). Cognitive-behavioral therapy versus other therapies: Redux. Clinical Psychology Review, 33(3), 395–405. https://doi.org/10.1016/j.cpr.2013.01.004

Barth, J., Munder, T., Gerger, H., Nüesch, E., Trelle, S., Znoj, H., Jüni, P., & Cuijpers, P. (2013). Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network meta-analysis. PLOS Medicine, 10(5), Article e1001454. https://doi.org/10.1371/journal.pmed.1001454

Berman, J., & Reich, C. (2010). Investigator allegiance and the evaluation of psychotherapy outcome research. European Journal of Psychotherapy & Counselling, 12(1), 11–21. https://doi.org/10.1080/13642531003637775

Cuijpers, P., Ebert, D. D., Acarturk, C., Andersson, G., & Cristea, I. A. (2016). Personalized psychotherapy for adult depression: A meta-analytic review. Behavior Therapy, 47(6), 966–980. https://doi.org/10.1016/j.beth.2016.04.007

Cuijpers, P., Karyotaki, E., Reijnders, M., & Ebert, D. D. (2018). Was Eysenck right after all? A reassessment of the effects of psychotherapy for adult depression. Epidemiology and Psychiatric Sciences, 28(1), 21–30. https://doi.org/10.1017/S2045796018000057

Cuijpers, P., Karyotaki, E., Reijnders, M., & Ebert, D. D. (2019). Is psychotherapy effective? Pretending everything is fine will not help the field forward. Epidemiology and Psychiatric Sciences, 28(3), 356–357. https://doi.org/10.1017/S204579601800080X

Cuijpers, P., van Straten, A., Andersson, G., & van Oppen, P. (2008). Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. Journal of Consulting and Clinical Psychology, 76(6), 909–922. https://doi.org/10.1037/a0013075

Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P., & Turner, E. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLOS ONE, 10(9), Article e0137864. https://doi.org/10.1371/journal.pone.0137864

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16(5), 319–324. https://doi.org/10.1037/h0063633

Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517. [Volume and issue are inferred from the May 1978 date, and the page was read from a scanned copy. No database record could be found.]

Flückiger, C., Del Re, A. C., Wampold, B. E., & Horvath, A. O. (2018). The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4), 316–340. https://doi.org/10.1037/pst0000172

Flückiger, C., Rubel, J., Del Re, A. C., Horvath, A. O., Wampold, B. E., Crits-Christoph, P., Atzil-Slonim, D., Compare, A., Falkenström, F., Ekeblad, A., Errázuriz, P., Fisher, H., Hoffart, A., Huppert, J. D., Kivity, Y., Kumar, M., Lutz, W., Muran, J. C., Strunk, D. R., . . . Barber, J. P. (2020). The reciprocal relationship between alliance and early treatment symptoms: A two-stage individual participant data meta-analysis. Journal of Consulting and Clinical Psychology, 88(9), 829–843. https://doi.org/10.1037/ccp0000594

Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., Chen, P., Hunot, V., & Churchill, R. (2014). Waiting list may be a nocebo condition in psychotherapy trials: A contribution from network meta-analysis. Acta Psychiatrica Scandinavica, 130, 181–192. https://doi.org/10.1111/acps.12275 [Issue number could not be verified.]

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351(6277), 1037. https://doi.org/10.1126/science.aad7243

King’s College London. (2019). Enquiry into publications authored by Professor Hans Eysenck with Professor Ronald Grossarth-Maticek. https://www.kcl.ac.uk/assets/news-statements/hans-eysenck-enquiry-final-may-2019.pdf

Lee, C. W., de Jongh, A., Farrell, D., Meysner, L., Dominguez, S., & El-Leithy, S. (2026). A critique of the 2025 American Psychological Association clinical practice guideline for the treatment of posttraumatic stress disorder in adults: Underrating EMDR effectiveness. Journal of EMDR Practice and Research, 20, 1–11. https://doi.org/10.34133/jemdr.0034

Munder, T., Brütsch, O., Leonhart, R., Gerger, H., & Barth, J. (2013). Researcher allegiance in psychotherapy outcome research: An overview of reviews. Clinical Psychology Review, 33(4), 501–511. https://doi.org/10.1016/j.cpr.2013.02.002

Munder, T., Flückiger, C., Gerger, H., Wampold, B. E., & Barth, J. (2012). Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? A meta-analysis. Journal of Counseling Psychology, 59(4), 631–637. https://doi.org/10.1037/a0029571

Munder, T., Flückiger, C., Leichsenring, F., Abbass, A. A., Hilsenroth, M. J., Luyten, P., Rabung, S., Steinert, C., & Wampold, B. E. (2018). Is psychotherapy effective? A re-analysis of treatments for depression. Epidemiology and Psychiatric Sciences, 28(3), 268–274. https://doi.org/10.1017/S2045796018000355

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530. https://doi.org/10.1177/1745691612465253

Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760. https://doi.org/10.1037/0003-066X.32.9.752

Tolin, D. F. (2010). Is cognitive-behavioral therapy more effective than other therapies? A meta-analytic review. Clinical Psychology Review, 30(6), 710–720. https://doi.org/10.1016/j.cpr.2010.05.003

Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work. Routledge.

Zilcha-Mano, S. (2017). Is the alliance really therapeutic? Revisiting this question in light of recent methodological advances. American Psychologist, 72(4), 311–325. https://doi.org/10.1037/a0040435