summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
TrueCore Behavioral Solutions, LLC.
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Aggression Replacement Training MHOS/SAOS/IMH
SPEP Review Period: 07/01/23-06/30/24
SPEP Review Conducted: 07/15/24
Florida Department of Juvenile Justice
Report Date: 9/29/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual review is being conducted at Orange Youth Academy/Orlando Intensive Youth
Academy for the period of 07/01/23-06/30/24. The program uses Aggression Replacement
Training as a primary service. The program has a formal training protocol provided by Crystal
Johnson who is a formally trained contractor. There were six staff trained during the review
period, one clinical director, one licensed therapist and four non-licensed therapists. Two of the
non-licensed therapists facilitated the group during the review period. The clinical director or his
trained designee provided fidelity monitoring for the facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual Aggression Replacement Training. There is a formal facilitator’s
manual. The manual includes the lessons, script and instructions for the facilitator, the order the
lessons are to be delivered, implementation guidelines, and provides detailed instructions for the
delivery of each lesson. Every youth who enters the program is immediately enrolled in the
service based on their assigned living modules. The intervention was held on Tuesdays,
Wednesdays, and Thursdays from 2:00 p.m.-3:00 p.m. The program maintains the attendance
sheets in a SPEP binder labeled “Aggression Replacement Training.” Documentation of internal
fidelity monitoring, staff training, staff evaluations and all other SPEP related documentation is
maintained in a tabbed separate binder labeled “SPEP Binder”.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
An observation of the Aggression Replacement Training group took place virtually via Microsoft
Teams on Tuesday 07/16/24 from 4:00 p.m.-5:00 p.m. The group consisted of six male youth, one
facilitator, and one staff. The group was held in a classroom setting. The facilitator utilized the
Aggression Replacement Training manual throughout this group session. She was knowledgeable
about the material being presented to the group. The lesson was "Making a Complaint". The
facilitator used her own words to convey the information to the youth. She used effective
motivational interviewing skills to encourage the youth to share more about their personal
experiences and to assess for understanding. There were no behavioral issues.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation of Aggression Replacement Training for the annual review period of 07/01/23-
06/30/24 showed there was no facilitator turnover. The program has a process/protocol in place in
the event of staff turnover. The Administrative staff or other staff trained in the Aggression
Replacement Training curriculum would take over the group. The program completed four cohorts
of ART from 08/02/23-10/13/23, 11/22/23-02/13/24, and 02/29/24-05/21/24 (two groups). A new
group began on 7/16/24 and is currently ongoing. There were six to ten (10) youth in each of
those groups. There were short gaps from 10/14/23-11/21/23 (37 days), from 02/14/24-02/28/24
(14 days), and from 05/22/24-07/15/24 (52 days). These gaps were due to each youth in the
program already having ARTor they were in the ART intervention at that time.
An informal interview with the clinical director confirmed that if a facilitator should leave the
program, the programs goal would be to hire more staff and train them as appropriate so that
there would be no gaps in services to the youth. The program is also striving to overlap the
groups and have a co-facilitator when possible.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program did have an in house process to perform internal fidelity monitoring. Fidelity
monitoring is to be performed by the clinical director or his trained designee. During the review
period, there were six staff trained in the Aggression Replacement Training curriculum, one
clinical director, one licensed therapist and four non-licensed therapists. There were (16) fidelity
checks conducted by the current clinical director. Monitoring was missed for the months of 07/23
and 10/23. This was due to the prior Clinical Director leaving and staffing issues. The program's
plan per the clinical director is to continue to overlap future groups to avoid gaps in services as
well as train other eligible staff in the curriculum so that they can both facilitate the group if
needed and provide the required fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Aggression Replacement Training. A review of the
documentation provided as well as an interview with the clinical director showed there had been
no corrective action taken during the review period. If facilitators are in need of corrective action
related to their delivery of the service, they would then receive progressive discipline in the form of
coaching, additional training, and formal write-ups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The clinical director and the licensed therapist both had their annual reviews as well as two non-
licensed therapists during the review period. The other two non-licensed therapists had ninety
(90) day reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 12 youth
High = 8 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 28%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 8 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (26 of 29) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 28% (8 of 29) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 18 25 72%
Totals 93 100 93%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Orange Youth Academy/Orlando Intensive Youth Academy.
65 youth completed the program during the review period. Of those youth, 29 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 100 points for Amount of Service: Duration. Of the 29 total youth sampled,
only 29 included dosage with end dates in the EBS Module. Of those youth with correct dosage,
29 received at least the recommended weeks of service. Youth in the sample completed between
11 and 12 weeks of service, with an average of 11.20 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, only 29 included dosage in the EBS Module. Of those youth with correct dosage, 29
received at least the recommended hours of service. Youth in the sample completed between 30
and 30 hours of service, with an average of 30 hours.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Quality of
Service Delivery score by ensuring fidelity monitoring is completed every month for each
facilitator.
Orange Youth Academy/Orlando Intensive Youth Academy can maintain their SPEP Amount of
Service score by continuing to ensure dosage for all youth is recorded accurately in EBS and by
continuing to ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

report_type: SPEP

file_name: FY2023-24 Orange Youth Academy-Orlando Intensive Youth Academy ART.pdf

program_name_raw: Orange Youth Academy - TrueCore

program_status_note: Closed

service_type: Aggression Replacement Training


state: FL

facility_name: Orange Youth Academy

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
SAOS
TrueCore Behavioral Solutions, LLC.
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Cannabis Youth Treatment
SPEP Review Period: 07/01/23-06/30/24
SPEP Review Conducted: 07/15/24
Florida Department of Juvenile Justice
Report Date: 9/29/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment .
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual review is being conducted at Orange Youth Academy/Orlando Intensive Youth
Academy for the period of 07/01/23-06/30/24. The program uses Cannabis Youth Treatment as a
primary service. The program has a formal training protocol provided by their regional clinical
director who is formally trained in the curriculum. There were six staff trained during the review
period, one clinical director, one licensed therapist and four non-licensed therapists. One licensed
therapist facilitated the group during the review period. The clinical director or his trained designee
provided fidelity monitoring for the facilitator.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual Cannabis Youth Treatment Series Volume I. There is a formal
facilitator’s manual. The manual includes the lessons, script and instructions for the facilitator, the
order the lessons are to be delivered, implementation guidelines, and provides detailed
instructions for the delivery of each lesson. Every youth who enters the program is immediately
enrolled in the service based on their assigned living modules. The intervention is held on
Wednesdays from 2:00 p.m.-3:00 p.m. The program maintains the attendance sheets in a SPEP
binder labeled “Cannabis Youth Treatment ”. Documentation of internal fidelity monitoring, staff
training, staff evaluations and all other SPEP related documentation is maintained in a tabbed
separate binder labeled “SPEP Binder”.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
An observation of the Cannabis Youth Treatment group took place virtually via Microsoft Teams
on Wednesday 07/17/24 from 2:30-3:30 p.m. The group consisted of nine male youth and one
facilitator. The facilitator used the Cannabis Youth Treatment manual throughout this group
session. She was knowledgeable about the material being presented to the group. The lesson
was Session 3: "Marijana Refusal." The facilitator used her own words to convey the information.
The facilitator used effective motivational interviewing skills to encourage the youth to share more
about their personal experiences and to assess for understanding. There were no behavioral
issues observed during this group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The staff were not all trained in the curriculum until 08/30/23. The program has a process/protocol
in place in the event of a turnover. The Administrative staff or other Cannabis Youth Treatment
trained staff would take over the group. Once training was completed, the program completed
three cohorts of CYT from 09/19/23-11/29/23, 11/30/23-02/14/24, and 02/22/24-05/29/24. There
was a gap from 05/30/24-07/16/24 (one month and sixteen (16) days). This was due to the
program waiting on enough new youth to begin a new cohort. A new cycle then began on
07/17/24 and is currently ongoing. There were between six to ten (10) male youth in each group.
An informal interview with the clinical director confirmed that if a facilitator should leave the
program, the programs goal would be to hire more staff and train them as appropriate so that
there would be no gaps in services to the youth. The program is also striving to overlap the
groups and have a co-facilitator when possible.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program did have an in house process to perform internal fidelity monitoring. This is to be
performed by the clinical director or his trained designee. During the review period there were six
staff trained in the Cannabis Youth Treament (CYT) curriculum, one clinical director, one licensed
therapist, and four non-licensed therapists. There were (16) fidelity checks conducted by the
current clinical director. Monitoring was missed for the months of 07/23 and 10/23. This was due
to the prior clinical director's leaving and staffing issues. The program's plan per the clinical
director is to continue to overlap future groups to avoid gaps in services as well as train other
eligible staff in the curriculum so that they can both facilitate the group if needed and provide the
required fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Impact of Crime. A review of the documentation provided as
well as an interview with the clinical director showed that there had been no corrective action
taken during the review period. If facilitators are in need of corrective action related to their
delivery of the service, they would then receive progressive discipline in the form of coaching,
additional training, and formal write-ups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The clinical director and licensed therapist both had their annual reviews on 12/19/23. Two non-
licensed therapists had 90 day reviews completed and two had annual reviews completed within
the review period. The other current staff that are trained and facilitating the Impact of Crime
intervention have not been employed by the program long enough for a review to be completed
yet.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 12 weeks for this type of service. Of the 12 youth in the
sample, 100% (12 of 12) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 12 youth in the
sample, 100% (12 of 12) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 6 youth
High = 6 youth
Total Youth in Sample = 12 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 12 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (12 of 12) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (6 of 12) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 100 100 100%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Orange
Youth Academy/Orlando Intensive Youth Academy.
65 youth completed the program during the review period. Of those youth, 12 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 12 total youth sampled,
only 12 included dosage with end dates in the EBS Module. Of those youth with correct dosage,
12 received at least the recommended weeks of service. Youth in the sample completed between
12 and 12 weeks of service, with an average of 12 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 12 total youth
sampled, only 12 included dosage in the EBS Module. Of those youth with correct dosage, 12
received at least the recommended hours of service. Youth in the sample completed between 12
and 12 hours of service, with an average of 12 hours.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Quality of
Service Delivery score by ensuring there are no unnecessary gaps in service and fidelity
monitoring is completed every month for each facilitator.
Orange Youth Academy/Orlando Intensive Youth Academy can maintain their SPEP Amount of
Service score by continuing to ensure dosage for all youth is recorded accurately in EBS and by
continuing to ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

file_name: FY2023-24 Orange Youth Academy-Orlando Intensive Youth Academy CYT.pdf

service_type: Cannabis Youth Treatment


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
MHOS/SAOS/IMH
TrueCore Behavioral Solutions, LLC.
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Impact of Crime
SPEP Review Period: 07/01/23-06/30/24
SPEP Review Conducted: 07/15/24
Florida Department of Juvenile Justice
Report Date: 9/29/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual review is being conducted at Orange Youth Academy/Orlando Intensive Youth
Academy for the period of 07/01/23-06/30/24. The program uses Impact of Crime (IOC) as a
primary service. This is a closed group. The program has a formal training protocol provided
either by DJJ staff or by a corporate trainer formally trained in the Impact of Crime curriculum.
There were eight staff trained during the review period, one director of case management and
seven case managers. Two case managers left the program in 11/23 after the cohort they were
running was completed. The director of case management and five case managers facilitated the
Impact of Crime group during the review period. The director of case management or her
designee provided fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Impact of Crime-Addressing the Harm curriculum. There is a formal
facilitator’s manual and separate student manuals.The manual includes the lessons, script and
instructions for the facilitator, the order the lessons are to be delivered, implementation guidelines,
and provides detailed instructions for the delivery of each lesson. Every youth who enters the
program is immediately enrolled in the service based on their assigned living modules. The
intervention is held on Mondays and Wednesdays, and Tuesdays and Thursdays from 6:00 p.m.-
7:00 p.m. The program maintains the attendance sheets in a SPEP binder labeled “Impact of
Crime Sign In Sheets.” Documentation of internal fidelity monitoring, staff training, staff
evaluations and all other SPEP related documentation is maintained in a tabbed separate binder
labeled “SPEP Binder.”
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of the Impact of Crime Group took place virtually via Microsoft Teams on Monday,
07/15/24 from 6:00-7:00 p.m. The group consisted of eleven (11) youth and one facilitator. The
facilitator utilized the Impact of Crime manual throughout this group session and was
knowledgeable about the material being presented to the group. The facilitator used her own
words to convey the information. This session began with an ice breaker, followed by a short
review of the previous lesson. The lesson today was "Hate Crimes". The facilitator utilized
effective motivational interviewing skills to encourage the youth to share more about their
personal experiences and to assess for understanding. There were no behavioral issues
observed during this group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation of Impact of Crime (IOC) for the annual review period of 07/01/23-
06/30/24 showed there was a short gap of seven days from 11/28/23-12/04/23 and a forty-three
(43) day gap from 12/05/23-01/17/24. The forty-three (43) day gap was due to the fact that the
youth at the program already had completed IOC or were in IOC at that time. The program
experienced some facilitator turnover for Impact of Crime. Two case managers left the program in
11/23; however, they did so after the cohort they were teaching ended, so there was no gaps in
services due to their exit. The program has a process/protocol in place in the event of a turnover.
The Administrative staff or other Impact of Crime trained staff would take over the group.
The program completed five cohorts of IOC from 08/07/23-11/27/23 (two groups), 10/01/23-
12/21/23 (one group), and 12/04/23-04/01/24 (two groups). There are four groups that are
currently ongoing which began on 01/17/24 (one group), 04/09/24 (two groups), and 07/12/24
(two groups). There were between two and eleven (11) male youth in each group.
An informal interview with the director of case management confirmed that if a facilitator should
leave the program, the programs goal would be to hire more staff and train them as appropriate
so that there would be no gaps in services to the youth. The program is also striving to overlap the
groups and have a co-facilitator when possible.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program did have an in house process to perform internal fidelity monitoring. This is to be
performed by the director of case management or her trained designee. The documentation
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
shows a total of seventeen (17) fidelity checks conducted by the current director of case
management, the prior acting director of case management and the facility administrator. Fidelity
monitoring was missed for the months of 07/23, 08/23 and 09/23. This was due to the prior
director of case management leaving and staffing issues. The program's plan per the current
director of case management is to continue to overlap future groups to avoid gaps in services as
well as training other eligible staff in the curriculum so that they can both facilitate the group if
needed and provide the required monthly fidelity monitoring for each facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Impact of Crime. A review of the documentation provided as
well as an interview with the director of case management showed that there had been no
corrective action taken during the review period. If facilitators are in need of corrective action
related to their delivery of the service, they would then receive progressive discipline in the form of
coaching, additional training, and formal write-ups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
There were two 90 day employee evaluations completed during the review period for the director
of case management and one case manager The other current staff that are trained and
facilitating the Impact of Crime intervention have not been employed by the program long enough
for a review to be completed yet.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Due to the Juvenile Justice Information System (JJIS) outage, this report does not contain data for
the period of March 2024 through June 2024. Data contained in this report is representative from
the period of July 2023 through February 2024.
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 64 youth in the
sample, 36% (23 of 64) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Due to the Juvenile Justice Information System (JJIS) outage, this report does not contain data for
the period of March 2024 through June 2024. Data contained in this report is representative from
the period of July 2023 through February 2024.
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 64 youth in the
sample, 53% (34 of 64) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 8 youth
Moderate-High = 24 youth
High = 28 youth
Total Youth in Sample = 64 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 28 youth
Total Youth in Sample = 64 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (60 of 64) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 44% (28 of 64) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 69 85 81%
This SPEP report evaluates Impact of Crime, an intervention delivered at Orange Youth
Academy/Orlando Intensive Youth Academy.
Due to the Juvenile Justice Information System (JJIS) outage, this report does not contain data for
the period of March 2024 through June 2024. Data contained in this report is representative from
the period of July 2023 through February 2024.
65 youth completed the program during the review period. Of those youth, 64 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 64 total youth sampled, only
45 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 23
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended weeks of service. Youth in the sample completed between 2
and 17 weeks of service, with an average of 12.75 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 64 total youth
sampled, only 61 included dosage in the EBS Module. Of those youth with correct dosage, 34
received at least the recommended hours of service. Youth in the sample completed between 1
and 33 hours of service, with an average of 21 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Quality of
Service Delivery score by ensuring there are no unnecessary gaps in service and fidelity
monitoring is completed every month for each facilitator.
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Amount of
Service score by ensuring dosage for all youth is recorded accurately in EBS and by ensuring
youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Orange Youth Academy-Orlando Intensive Youth Academy IOC.pdf

service_type: Impact of Crime