“Research Shows”: The Industry’s Own Studies

Who pays for the studies troubled teen programs cite, who publishes them, and what their own tables show.

7 min read Updated

Contents
  1. What we found
  2. “An independent research company”
  3. Paid for by the trade group
  4. The trade group’s own journal
  5. What the tables show
  6. “Significant positive results” at six months
  7. “High” follow-up rates
  8. Eight teenagers
  9. A link the table does not show
  10. What the studies do not measure
  11. A seal for collecting data
  12. How to read a “research shows” claim
  13. Sources

Program websites and brochures tell parents their treatment is “proven effective” and “evidence-based.” Much of the research behind those words is paid for, edited and published by the industry itself. And the studies’ own tables often say less than their summaries.

What we found #

  • Aspen Education Group told parents its outcome study was “conducted by an independent research company.” The study’s own report says: “Aspen Education Group provided funding for this study.”
  • The researcher who wrote that report later became chief editor of the trade group’s journal.
  • The research center behind most wilderness therapy studies took at least $1 million from the industry’s trade group.
  • A study in that journal reported “significant positive results” six months after discharge. It had six-month answers from 148 of the 1,389 youths it started with.
  • The trade group’s research seal certifies that a program collects data, not that its results are good. Programs may put it “in promotional materials.”
  • In the issue we read, “restraint,” “seclusion” and “runaway” do not appear once.

Part of our series How the Troubled Teen Industry Manages Its Reputation. Every example comes from a court record, a government report, the company’s own words or published reporting, and each is linked in the sources at the end. Where something is only alleged in a lawsuit, we say so.

“An independent research company” #

A brochure from Aspen Education Group, which owned troubled teen programs across the country, is headed “Aspen Education Group’s Residential Therapeutic Schools and Programs: Proven Effective.” It says the study was “conducted by an independent research company” and calls the results “groundbreaking” (our copy).

Families, the brochure says, “are generally living better lives today because of their decision to enroll in an Aspen program.”

The study’s first report was presented at the American Psychological Association’s 2006 convention by Ellen Behrens and Kristin Satterfield of Canyon Research & Consulting. Like the brochure, it covers teenagers admitted from August 2003, in a study approved by the same ethics board, the Western Institutional Review Board. It ends with a disclosure: “Aspen Education Group provided funding for this study” (our copy).

One of the nine programs studied was Mount Bachelor Academy in Oregon. In 2009 the state found “nine counts of abuse and neglect based on unusual therapeutic techniques,” the Bend Bulletin reported, including “sexualized role-playing in front of other students, sleep deprivation and extended manual labor as punishment.” Oregon ordered the school “to cease classes and therapy,” and Aspen closed it.

The report is also plain about its design: “like most outcome research in residential treatment, this study did not use a control group.” Without one, a study cannot tell whether teenagers improved because of the program or because time passed. The brochure’s “because of” is a claim the study cannot make.

Ten years later, Behrens became chief editor of the trade group’s own journal. “It is with great pleasure that I begin my tenure as Chief Editor,” she wrote in its 2016 issue (p. 4).

Wilderness programs often point to studies showing that their treatment works.

In 2020 Undark reported that the research center producing most of those studies had received at least $1 million from the Outdoor Behavioral Healthcare Council, the industry’s trade group, since 1999. All seven of its researchers had worked in the industry or had strong ties to it, and the studies appeared mainly in journals founded by related trade groups.

After the reporting, the center added a statement naming the OBH Council and NATSAP as its funders.

The trade group’s own journal #

NATSAP, the National Association of Therapeutic Schools and Programs, publishes the Journal of Therapeutic Schools and Programs. We read one full issue, Volume 8 (2016), which has ten articles. Our copy is in our document library.

Its masthead calls it “A Publication of the National Association of Therapeutic Schools and Programs.” The editorial board includes the heads of member programs: Jared Balmer of Island View, Dustin Tibbits of New Haven Residential Treatment Center, John Santa of Montana Academy, and Robert Cooley of Catherine Freer Wilderness, where 15-year-old Erica Harvey died in 2002 (her mother’s testimony to Congress).

One person decides what runs: “Dr. Behrens will make final decisions regarding publication status” (p. 98).

The journal’s rules say authors “are required to reveal any possible conflict of interest,” such as “funding provided by program” (p. 97). No article in the issue prints one.

Yet three of its outcome studies were written by staff of the program being studied. Two come from Evoke Therapy Programs, by its research director and by its clinical director and partner. The third is by the clinical director and the research director of Pine River Institute, the program it evaluates (author notes).

What the tables show #

Each outcome study in the issue measures teenagers when they arrive and again later, with no comparison group. A methods article in the same issue explains what that means: “we can measure change but cannot conclude that the treatment program or intervention caused the change” (p. 15).

“Significant positive results” at six months #

A study of wilderness programs in NATSAP’s database reports “clinically and statistically significant positive results” at intake, discharge and six months after discharge (p. 24).

It started with 1,389 youths. “While 1,014 youth had pre and post matched pairs of data, only 14.6% of those pairs had data for six months post discharge,” the authors write, calling it one of the study’s “major limitations” (p. 36). The youth six-month results rest on 148 young people (p. 41).

Mothers were the largest group of parents who answered. At six months, their average rating of their children was 49.70 (p. 42), above the questionnaire’s clinical cutoff of 47 (p. 27).

“High” follow-up rates #

A second study says it “achieved high 6-month follow-up response rates.” Its table shows answers from 44% of young adults and 55% of adolescents (p. 23).

Most of those teenagers were not home. Eighty-three percent of the adolescents went straight from the wilderness program to “a residential treatment center, boarding school, or another type of aftercare (AC) program” (p. 19).

And of the adolescents’ six-month questionnaires, “AC programs administered 70.5%” (p. 19). Most of the “post-discharge” answers were collected by staff of the next program the child had been sent to.

Eight teenagers #

A third study, of one wilderness program, reports “clinically and statically significant change.” Of 332 eligible teenagers, 118 took part, and only those who stayed at least five weeks were eligible (p. 48).

“Response rates at the 6-month follow-up were low”: 8 teenagers and 10 parents (p. 49). The parents’ average score, 56.4, was above the cutoff the study itself uses for “the community range of functioning,” a score below 47 (p. 48).

A fourth study’s summary says parents’ skills are “quantitatively associated with” their children’s substance use after treatment (p. 57).

Its own results say the opposite: the parent-skill measures “were not significant predictors of post-treatment substance use” (p. 61). The authors also write that their design “does not allow us to claim that our results are caused by our treatment” (p. 63).

What the studies do not measure #

Every outcome the studies measure for young people is a score for symptoms, family functioning or satisfaction. None counts restraints, time in isolation, injuries, runaways or abuse reports; across the whole issue, the words “restraint,” “seclusion” and “runaway” do not appear.

NATSAP’s ethical principles, printed in the same issue, commit members to “fully disclose to prospective candidates the nature of services, benefits, risks and costs” (p. 102). Research that never looks for harm gives a program no risks to disclose.

The issue’s last article proposes “practice standards” for NATSAP programs. Its evidence is 11 case studies recruited “through announcements at NATSAP conferences,” and the authors note that the reports “focused on model cases with positive outcomes” (p. 92).

For “confrontational attitudes,” which it says occurred “especially since many of the students were placed in treatment by their parents and not of their own accord,” it suggests wilderness “solo” interventions, in which a teenager spends long periods alone (p. 93).

A seal for collecting data #

The same issue describes its “Research Designated Program” seal, meant to “express to consumers that not all programs are alike, and those who honestly examine their work” (p. 8).

To earn it, a program must collect data from 70% of the teenagers it enrolls and 50% at discharge. For after discharge there is only a “target goal” of 20 to 50%, and a program that misses it keeps the seal by “submitting a review of their data collection efforts and a plan” (p. 10).

Nothing requires a program to publish its results, or to have good ones. Once approved, “you will be allowed to include the RDP seal on your website and in promotional materials” (p. 10).

NATSAP’s own description of its database admits it “has relied in large part on a handful of programs who have contributed the majority of data” (p. 8).

How to read a “research shows” claim #

  • Who paid for it? Look for a funding or disclosure statement, and ask the program.
  • Who published it? A trade group’s journal is not an independent one.
  • Was there a comparison group? Without one, a study cannot show the program caused the change.
  • How many answered at follow-up, and where were they? A result from a small share of families, many of them still in another program, says little about the rest.
  • Did it count harm? Ask for restraint, seclusion, injury and runaway numbers. Check the state’s inspection reports.

Sources #