summary: SPEP - FY25-26

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
MHOS/SAOS/INT
Youth Opportunity Investments, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Impact of Crime
SPEP Review Period: 09/01/2024-08/31/2025
SPEP Review Conducted: 09/03/2025; 09/08/2025
Florida Department of Juvenile Justice
Report Date: 09/03/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual SPEP review was conducted at Orange Youth Academy/Orlando Intensive Youth
Academy for the period of 09/01/2025-08/31/2025. The documentation shows the program utilizes
Impact of Crime (IOC) as a primary service. The program has a formal training protocol provided
either by Department of Juvenile Justice staff or by a corporate trainer formally trained in the IOC
curriculum. There were nine staff trained during the review period, including two case managers,
two transition services managers, one career and technical education (CTE) manager, one
assistant facility administrator, one unit manager, one staff development specialist, and one
director of case management. One case manager left the program after the cohort she was
running had completed. The director of case management, two case managers, the CTE manager
and two transition services managers facilitated the IOC group during the review period.
The program can earn 1 point if all facilitators are trained in the primary service prior to facilitating
groups.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
A review of the documentation for the annual review period of 09/01/2024-08/31/2025 shows the
program utilizes the Impact of Crime: Addressing the Harm v 4.0 as the manual. This fidelity
support specialist reviewed the manual. The facilitator’s manual is divided into seven chapters, an
appendix with handouts, and the student guide section (which includes the student exercises).
The manual addresses the order in which to deliver the content and gives the facilitators an
outline to follow. The facility does not have separate student workbooks to give out. The
facilitators make copies of the exercises before each session to hand out to the youth. The group
is a closed group with no more than twelve youth in each cohort. The curriculum is to be delivered
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
to the youth over a minimum of sixteen (16) weeks for thirty-two (32) hours. The program keeps
all documentation in a binder labeled "IOC SPEP" in the case management office. All youth who
are admitted to the program receive the primary service. The intervention is held on Mondays and
Wednesdays from 12-1 p.m. The program maintains the attendance sheets in a SPEP binder
labeled “Impact of Crime Sign In Sheets.” Documentation of internal fidelity monitoring, staff
training, staff evaluations and all other SPEP related documentation is maintained in a tabbed
separate binder labeled “SPEP Binder.”
The program can earn 2 points if the manual/protocol is detailed to the extent it explains how each
specific session is delivered and there is a script or outline for each session.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of the Impact of Crime (IOC) group took place virtually by Zoom on 09/08/2025
from 12-1 p.m. The group consisted of eleven (11) male youth and one facilitator. The group was
held in a classroom setting. The facilitator utilized the IOC manual throughout this group session
and was knowledgeable about the material being presented to the group. The facilitator used her
own words to convey the information. This session began with an ice breaker, followed by a short
review of the previous lesson. A copy of the lesson plan was provided to the reviewer prior to the
observation. The lesson presented was Chapter #6 "Managing Conflict." The discussion for this
group was hate crimes.The facilitator utilized effective motivational interviewing skills to
encourage the youth to share more about their personal experiences and to assess for
understanding. She ensured that each youth understood the topics presented before she moved
on in the lesson.
The program can earn 1 point if the program follows the manual/protocol when delivering the
primary service.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did experience some turnover during the review period. Two case managers left the
program in 05/2025 and 08/2025. They were both facilitating group when they exited, however,
those groups were transferred to two other trained co-facilitators with no gaps in the groups.
There were no other gaps in the service of Impact of Crime (IOC) to the youth during the review
period. In the event of turnover, administrative staff or other trained staff would facilitate the
groups.
The program completed seven cohorts of IOC: 07/12/2024-11/08/2024 with ten (10) youth and
two staff, 07/12/2024-11/15/2024 with ten (10) youth and two staff, 11/08/2024-02/25/2025 with
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
ten (10) youth and two staff, 01/06/2025-05/07/2025 with ten (10) youth and two staff,
01/06/2025-05/14/2025 with ten (10) youth and two staff, 03/05/2025-06/30/2025 with eight youth
and two staff, 03/05/2025-06/25/2025 with nine youth and two staff. One case manager (CM) left
during group, the transition services manager took over as the facilitator. There are two ongoing
groups at the time of this review. One group began on 06/09/2025 with ten (10) youth and two
staff. One facilitator left on 07/25/2025 and the co-facilitator took over the group on 08/20/2025.
The other ongoing group began on 08/02/2025 with ten (10) youth and two staff.
An informal interview with the director of case management confirmed if a facilitator should leave
the program, the program's goal would be to hire more staff and train them as appropriate so that
there would be no gaps in service to the youth. The program is also striving to overlap the groups
and have a co-facilitator when possible.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the annual review period from 09/01/2024-08/31/2025 shows
the program did have an in-house process to perform internal fidelity monitoring. This is to be
performed by the director of case management or the trained designee. Fidelity checks were
completed by the director of case management monthly for each facilitator. No fidelity monitoring
was missed during the review period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Impact of Crime. A review of the documentation provided as
well as an interview with the director of case management showed there had been no corrective
action taken during the review period. If facilitators are in need of corrective action related to their
delivery of the service, they would then receive progressive discipline in the form of coaching,
additional training, and formal write-ups.
The program can earn 1 point if any corrective action needed is identified, applied and
demonstrated.
At the time of the review, the program met the criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four facilitators' employee evalutions found an entry for evaluation of their
performance in delivering Impact of Crime. These included comments on any strengths or
weaknesses of which the facilitator should be aware. The fifth facilitator is new and has not
received an evaluation to date.
The program can earn 1 point if each facilitator's performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program met the criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 26 youth in the
sample, 100% (26 of 26) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 26 youth in the
sample, 100% (26 of 26) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 10 youth
Moderate-High = 9 youth
High = 5 youth
Total Youth in Sample = 26 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 19%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 5 youth
Total Youth in Sample = 26 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (24 of 26) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 19% (5 of 26) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 13 25 52%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Orange Youth
Academy/Orlando Intensive Youth Academy.
27 youth completed the program during the review period. Of those youth, 26 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 26 total youth sampled, all
26 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 26
received at least the recommended weeks of service. Youth in the sample completed between 16
and 18 weeks of service, with an average of 17.27 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Contact Hours. Of the 26 total youth
sampled, all 26 included dosage in the EBS Module. Of those youth with correct dosage, 26
received at least the recommended hours of service. Youth in the sample completed between 31
and 32 hours of service, with an average of 31.73 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can maintain their SPEP Quality of
Service Delivery score by continuing the practices in place at the time of this review.
Orange Youth Academy/Orlando Intensive Youth Academy can maintain their SPEP Amount of
Service score by continuing to ensure dosage for all youth is recorded accurately in EBS and by
continuing to ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

report_type: SPEP

file_name: FY2025-26 OYA-OIYA IOC.pdf

program_status_note: Open

service_type: Impact of Crime


state: FL

facility_name: Orlando Intensive Youth Academy

summary: SPEP - FY25-26

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
MHOS/SAOS/INT
Youth Opportunity Investments, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Living In Balance -SAOS
SPEP Review Period: 09/01/2024-08/31/2025
SPEP Review Conducted: 09/03/2025; 09/08/2025
Florida Department of Juvenile Justice
Report Date: 09/03/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or
interpersonal problems or issues faced by an individual and involves a group of youths interacting
with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual SPEP review was conducted for Orange Youth Academy/Orlando Intensive Youth
Academy for the dates of 09/01/2024-08/31/2025. The documentation shows the program utilizes
Living In Balance (LIB) as a primary service. The program has formal online training for this
intervention which is conducted under the direction of the clinical director. There are currently six
staff trained in LIB, including two licensed mental health therapists and four non-licensed
therapists.
The program can earn 1 point if all facilitators are trained in the primary service prior to facilitating
groups.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
A review of the documentation for the annual review period of 09/01/2024-08/31/2025 shows the
program utilizes the curriculum manual called "Living in Balance: Moving from a Life of Addiction
to a Life of Recovery." The fidelity support specialist reviewed the manual. The manual addresses
the order in which to deliver the content and gives the facilitators an outline to follow. The program
does not have separate student workbooks to give out. The facilitators make copies of the
exercises before each session to hand out to the youth.
The primary service is an open group that is held year-round. The program provides Living in
Balance (LIB) on Mondays and Wednesdays from 3:30-4:30 p.m. The program keeps all
documentation in a binder labeled "LIB SPEP" in the clinical director’s office. All SAOS youth who
are admitted to the program receive the primary service. This is an open, ongoing group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The program can earn 2 points if the manual/protocol is detailed to the extent that it explains how
each specific session is delivered and there is a script or outline for each session.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Living In Balance (LIB) took place virtually by Zoom on Monday 09/08/2025
from 4-5 p.m. The group consisted of eight male youth and one facilitator. The group was held in
a classroom setting. The facilitator utilized the LIB manual throughout this group session and was
knowledgeable about the material being presented to the group. The facilitator used her own
words to convey the information. This session began with an ice breaker, followed by a short
review of the previous lesson. The lesson was Skill #9 "Reducing Stress." The facilitator utilized
effective motivational interviewing skills to encourage the youth to share more about their
personal experiences and to assess for understanding. She ensured the youth understood each
topic presented prior to moving on the lesson.
The program can earn 1 point if the program follows the manual/protocol when delivering the
primary service.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did not experience any facilitator turnover during the review period. An informal
interview with the clinical director confirmed if a facilitator should leave the program,
administration or other trained facilitator would take over the group. The program's goal would
then be to hire more staff and train them as soon as possible so there would be no gaps in service
to the youth. The program is also striving to overlap the groups and have a co-facilitator when
possible to avoid any gaps in service to the youth. There were three cohorts of Living in Balance
completed from 09/01/2024-02/25/2025 with twelve (12) youth and two staff in each group. Three
new cycles began on 03/03/2025 and will end on 09/08/2025 with eleven (11) youth and two staff
in each group.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A review of the documentation for the period from 09/01/2024-08/31/2025 shows the program did
have an in-house process to perform internal fidelity monitoring. Monitoring is to be performed by
the clinical director or a trained designee. Five therapists that facilitated group during this review
period. Fidelity monitoring was completed monthly for each facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Living In Balance. A review of the documentation provided
as well as an interview with the clinical director confirmed there had been no corrective action
taken during the review period. If facilitators are in need of corrective action related to their
delivery of the service, they would then receive progressive discipline in the form of coaching,
additional training, and formal write-ups
The program can earn 1 point if any corrective action needed is identified, applied and
demonstrated.
At the time of the review, the program met the criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of five facilitators' employee evalutions found an entry for evaluation of their performance
in delivering Living In Balance. These included comments on any strengths or weaknesses of
which the facilitator should be aware. One facilitator is new and has not received an evaluation to
date.
The program can earn 1 point if each facilitator's performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program met the critieria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 5 youth
Moderate-High = 2 youth
High = 2 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 22%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 2 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (9 of 9) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 22% (2 of 9) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 17 25 68%
Totals 67 95 71%
This SPEP report evaluates Living In Balance, an intervention delivered at Orange Youth
Academy/Orlando Intensive Youth Academy.
27 youth completed the program during the review period. Of those youth, 9 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 9 total youth sampled, all 9
included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 6
and 16 weeks of service, with an average of 12.22 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 9 total youth sampled,
all 9 included dosage in the EBS Module. Of those youth with correct dosage, 0 received at least
the recommended hours of service. Youth in the sample completed between 8 and 26 hours of
service, with an average of 17.44 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 79%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Quality of
Service Delivery score by can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Amount of
Service score by ensuring dosage for all youth is recorded accurately in EBS and by ensuring
youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

file_name: FY2025-26 OYA-OIYA LIB.pdf

service_type: Living in Balance


summary: SPEP - FY25-26

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
MHOS/SAOS/INT
Youth Opportunity Investments, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32829
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 09/01/2024-08/31/2025
SPEP Review Conducted: 09/03/2025; 09/04/2025
Florida Department of Juvenile Justice
Report Date: 09/03/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An annual SPEP review is being conducted at Orange Youth Academy/Orlando Intensive Youth
Academy for the period of 09/01/2024-08/31/2025. The Skillstreaming intervention is being
evaluated. The documentation shows the program utilizes Skillstreaming as a primary service.
This is an open group. The program has a formal training protocol. Staff are required to complete
online training. All therapists are required to complete facilitator training as part of the onboarding
process. This is overseen and provided by the clinical director. There were six staff trained during
the review period, two licensed therapists and four non-licensed therapists. During the review
period, one licensed therapist and three non-licensed therapists faciltated the group. The clinical
director and one licensed therapist provided fidelity monitoring.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Skillstreaming the Adolescent curriculum for all youth assigned to the
program. There are no student manuals for this service. The curriculum includes a facilitator’s
manual and skill cards for the youth. This service is provided three times a week and groups are
run year-round with no more than ten (10) youth per group. Skillstreaming is an open group. The
intervention is held on Tuesdays and Thursdays from 5-6 p.m. The program maintains their
records in SPEP binders by intervention. The binder is then tabbed by individual sections. All IMH
and MHOS youth are required to complete Skillstreaming when entering the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The program can earn 2 points if the manual/protocol is detailed to the extent it explains how each
specific session is delivered and if there is a script or outline for each session.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of the Skillstreaming group took place virtually by Zoom on 09/04/2025 from 5-6
p.m. The group consisted of ten (10) male youth and one facilitator. The group was held in a
classroom setting. The facilitator utilized the Skillstreaming manual throughout this group session
and was knowledgeable about the material being presented to the group. The facilitator used her
own words to convey the information. This session began with an ice breaker, followed by a short
review of the previous lesson. The lesson was Lesson #3 "Having a Conversation." The facilitator
utilized effective motivational interviewing skills to encourage the youth to share more about their
personal experiences and to assess for understanding. She ensured that the youth understood
each topic presented prior to moving on in the lesson.
The program can earn 1 point if the program follows the manual/protocol when delivering the
primary service.
At the time of the review, the program met the criteria to earn a score of 1.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did not experience any facilitator turnover during the review period. The program's
protocol in the event of turnover is that administrative or other trained staff would take over the
groups. An informal interview with the clinical director confirmed that if a facilitator should leave
the program, the program's goal would be to hire more staff and train them as appropriate so that
there would be no gaps in services to the youth. The program is also striving to overlap the
groups and have a co-facilitator when possible. There were no gaps in service of Skillstreaming to
the youth.
Four cohorts were conducted concurrently and all completed on 02/25/2025. The groups included
nine youth and one staff; eight youth and one staff; ten (10) youth and one staff; and five youth
and one staff. Three new cohorts began on 03/04/2025 with ten (10) youth in each group.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the annual review period from 09/01/2024-08/31/2025 provided
that the program did have an in-house process to perform internal fidelity monitoring. This is
performed by the clinical director or the trained designee. Four staff facilitated Skillstreaming who
are one licensed therapist and three non-licensed therapists. Fidelity monitoring was completed
monthly for each facilitator. No fidelity monitoring was missed during the review period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program met the criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Skillstreaming. A review of the documentation provided as
well as an interview with the clinical director showed that there had been no corrective action
taken during the review period. If facilitators are in need of corrective action related to their
delivery of the service, they would receive progressive discipline in the form of coaching,
additional training, and formal write-ups
The program can earn 1 point if any corrective action needed is identified, applied and
demonstrated.
At the time of the review, the program met the criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four facilitators' annual employee evalutions found an entry for evaluation of their
performance in delivering Skillstreaming. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
The program can earn 1 point if each facilitator's performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program met the criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 17 youth in the
sample, 76% (13 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 17 youth in the
sample, 88% (15 of 17) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 7 youth
High = 3 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 18%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 3 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (15 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 18% (3 of 17) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 13 25 52%
Totals 67 85 79%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Orange
Youth Academy/Orlando Intensive Youth Academy .
27 youth completed the program during the review period. Of those youth, 17 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 17 total youth sampled, all
17 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 10
and 25 weeks of service, with an average of 17.29 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended hours of service. Youth in the sample completed between 19
and 48 hours of service, with an average of 33.47 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can maintain their SPEP Quality of
Service Delivery score by continuing the practices in place at the time of this review.
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Amount of
Service score by ensuring dosage for all youth is recorded accurately in EBS and by ensuring
youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2025-26 OYA-OIYA Skillstreaming.pdf

service_type: Skillstreaming the Adolescent