summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Thinking for A Change
SPEP Review Period: 01/01/22-12/31/22
SPEP Review Conducted: 01/31/23
Florida Department of Juvenile Justice
Report Date: April 10, 2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Thinking for A Change (T4C) is being reviewed as a primary service for the dates of 01/01/22-
12/31/22. There is a formal training protocol -five days, twenty (20) hours- for this intervention
which staff must attend, facilitated by a certified Thinking for a Change trainer. The training
consists of reviewing and learning the curriculum manual and conducting mock groups. During the
review period, there were eight staff formally trained in the T4C curriculum: one Facility
Administrator, one Director of Case Management, two Transition Service Managers, and four
Case Managers. Four Case Managers and two Transition Services Managers facilitated group
during the review period. The Director of Case Management and the Facility Admistrator provided
fideity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the brand name manual Thinking for A Change, version 3.1. There is a
facilitator manual only. Youth are provided photocopied handouts for each lesson. The manual
provides a script and instructions for twenty-five (25) lessons. The curriculum also includes
implementation guidelines, identifies the order the lessons should be delivered and includes
detailed instructions for the delivery of each lesson. All youth receive the Thinking for a Change
(T4C) intervention as a part of their programming.
The program maintains all SPEP documentation in a tabbed binder separated into sections and
then organized by intervention. The T4C group is held on Mondays, Wednesdays, and Fridays
from 2-3pm and 3-4pm.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for A Change was observed via Microsoft Teams on Friday, 02/10/23 from
2pm to 3pm. There were three youth and one staff present in the group. The facilitator provided a
copy of lesson 25, Next Steps. The facilitator followed the lesson plan according to the manual,
did not read the lesson verbatim but rather used her own words to convey the meaning of the
discussed topic. She was very easy for the youth to follow. The facilitator adhered to the lesson
plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation provided and an interview with the Director of Case Management
confirmed the program has had turnover during the review period; two Case Managers and one
Transition Services Manager terminated their employment. One Case Manager was promoted to
another position within the progam. One Case Manager left after the cohort of Thinking for a
Change (T4C) she had been facilitating was completed. The other Case Manager left during
cohorts and the groups transferred to another facilitator without a gap in service.
There is currently one Case Manager and one Director of Case Management facilitating T4C.
There were four cohorts of T4C completed during the review period. The four completed cohorts
ran 12/09/21-03/10/22, 04/11/22-06/19/22, 06/20/22-10/05/22 and 09/09/22-12/23/22. There are
two groups ongoing which began on 10/19/22 and 10/31/22.
The program's policy regarding anticipated turnover is addressed by having administrative or
other trained staff step in to facilitate the group, if necessary.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Thinking for a Change (T4C). The monitoring was completed by the Director of Case
Management or the Facility Administrator, who is formally trained in the curriculum. The program
uses internal fidelity monitoring checklists specific to T4C to document the monitoring. The
completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Monitoring for T4C". Each tab had a copy of the fidelity checklist completed on the facilitator(s)
who perfomed groups during the specific month.
The program completed internal fidelity monitoring for all facilitators for each month they facilitated
a T4C group, except for the month of March 2022.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of Thinking for a Change (T4C). The program uses internal fidelity
montoring/corrective action checklists specific to the curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or, if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Case Management and a review of the documentation provided
verified the program did not take any corrective actions during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators confirmed these staff received
either an annual evaluation or ninety (90)-day evaluation within the last year. The performance
evaluations for these staff found entries of the staff's skill in delivering Thinking for A Change.
These included comments on any strengths or weaknesses of which the facilitatorshould be
aware. The new Transition Services Manager has not been employed long enough to receive an
evaluation and he has not facilitated any groups to date.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 33 youth in the
sample, 85% (28 of 33) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 45 contact hours for this type of service. Of the 33 youth in the
sample, 82% (27 of 33) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 15 youth
High = 13 youth
Total Youth in Sample = 33 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 13 youth
Total Youth in Sample = 33 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (31 of 33) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 39% (13 of 33) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 94 100 94%
This SPEP report evaluates Thinking for A Change, an intervention delivered at Orange Youth
Academy.
39 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 33 total youth sampled, only
32 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 28
received at least the recommended weeks of service. Youth in the sample completed between 1
and 18 weeks of service, with an average of 16 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 33 total youth
sampled, all 33 included dosage in the EBS Module. Of those youth, 27 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 3 and 49 hours of
service, with an average of 43 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Orange Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and fidelity monitoring is completed every month for
each facilitator.
Orange Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 OYA T4C.pdf