summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Mental Health Overlay Services /
Substance Abuse Overlay Services
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimmee, Florida 34744
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 05/01/20-04/30/21
SPEP Review Conducted: 05/25/21
Florida Department of Juvenile Justice
Report Date: 8/25/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There were nine staff trained in the Skillstreaming curriculum- one Clinical Director, one Assistant
Clinical Director, and seven Therapists. Three Therapists facilitated this intervention for the
Mental Health Overlay/Substance Abuse Overlay youth during the review period. All staff received
formal training from a qualified trainer in the Skillstreaming the Adolescent curriculum.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided for review was Skillstreaming the Adolescent, A Guide for Teaching
Prosocial Skills, Third Edition. The facilitator manual contains a script for the facilitator for the
groups. It provides lessons and includes implementation guidelines, identifies the order the
lessons should be delivered in and includes detailed instructions for the delivery of each lesson.
Skillstreaming The Adolescent is an open/ongoing group. There was one cohort throughout the
review period.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Skillstreaming the Adolescent group was observed via Zoom on 06/01/21 from 2:30-3:30 by
S. Wilson, TAS. There was 1 Staff, 1 Facilitator and 10 youth in the group. The facilitator provided
a copy of the lesson plan(Lesson 16-Expressing Your Feelings) to the observer from the program
manual. The facilitator followed the lesson according to the manual, did not read the lesson
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
verbatim and used her own words to convey the lesson to the group. She was easy to follow and
adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The facility has experienced facilitator turnover during the review period. However, there were no
gaps in services in the Skillstreaming the Adolescent intervention to the youth during the review
period. The remaining Therapists and Assistant Clinical Director continued with the groups.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process to monitor the delivery of Skillstreaming the Adolescent . Internal
fidelity monitoring was missing for the months of 05/20, 06/20, 08/20, 01/21, 02/21, 03/21 and
04/21 for Mental Health Overlay/Substance Abuse Overlay unit.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the Director of Case Management verified the program has a process for
corrective actions based on the fidelity monitoring conducted. The program uses internal fidelity
monitoring/corrective action checklists specific to Skillstreaming the Adolescent. An interview with
the Director of Case Management and review of the documentation provided verified there were
no corrective actions noted nor taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The performance evaluations of all trained facilitators reviewed and included discussion of skill in
delivering Skillstreaming the Adolescent.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 23 youth in the
sample, 100% (23 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 23 youth in the
sample, 100% (23 of 23) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 8 youth
High = 11 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 11 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (21 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 48% (11 of 23) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 83 85 98%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services.
24 youth completed the program during the review period. Of those youth, 23 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 23 total youth sampled, all
23 included dosage with end dates in the EBS Module. Of those youth, 23 received at least the
recommended weeks of service. Youth in the sample completed between 26 and 71 weeks of
service, with an average of 49 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 10 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, all 23 included dosage in the EBS Module. Of those youth, 23 received at least the
recommended hours of service. Youth in the sample completed between 45 and 50 hours of
service, with an average of 50 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Quality of Service Delivery score by ensuring fidelity monitoring
is completed every month for each facilitator.
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Amount of Service score by ensuring that dosage for all youth
is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8

file_name: FY2020-21 KYA MHOS Skillstreaming.pdf


state: FL

facility_name: Kissimmee Youth Academy

summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Mental Health Overlay Services /
Substance Abuse Overlay Services
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimmee, Florida 34744
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 05/01/21-04/30/22
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming intervention is being reviewed. There is a training protocol conducted by the
Director of Clinical Services or designee to include training in the Skillstreaming the Adolescent
currciulum, group observations, and debriefings. The program maintains all SPEP documentation
in a tabbed binder separated into sections and organized by intervention. During the review period
from 05/01/21 to 04/30/22 there were two licensed Therapists and seven non-licensed Therapists
trained in the curriculum. Seven non-licensed Therapists facilitated group during the review
period. The two licensed Therapists provided fidelity monitoring for the non-licensed Therapists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual in use is Skillstreaming the Adolescent, A Guide for Teaching Prosocial Skills, Third
Edition. There is a facilitator manual. The facilitator manual contains a script for the facilitator for
the groups. It provides lessons and includes implementation guidelines, identifies the order the
lessons should be delivered in and includes detailed instructions for the delivery of each lesson.
The intervention is held on Tuesdays and Thursdays from 2:30 to 3:30 and 3:30 to 4:30 . The
program maintains their records in SPEP binders which are divided by intervention and BDD/DD,
MHOS, or SAOS. All youth at Kissimmee Youth Academy receive Skillstreaming. Skillstreaming is
an open, ongoing group. There are up to thirteen (13) youth in each ongoing group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed on 05/03/22 from 11:25 to 12:25p.m.
There were two staff and nine youth present in the group. The facilitator provided a copy of the
the lesson "How to Communicate and Dealing with Contradicting Messages" to the reviewer. The
facilitator followed the lesson plan according to the manual, did not read the lesson verbatim but
rather used his own words to convey the meaning of the discussed topic. He was very
knowledgeable of the materials. He was easy for the youth to follow. He ensured the youth
understood the oncepts prior to moving forward in the lesson. The facilitator adhered to the lesson
plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Clinical
Services confirmed the program has had some turnover during the review period. The program
lost one Assistant Clinical Director and four Therapists during the review period. They have since
hired new staff and trained them. The Skillstreaming the Adolescent groups being facilitated by
the therapists who left were transitioned to other trained Therapists without a gap in service. The
program's policy regarding anticipated turnover is addressed by having administrative or other
trained staff step in and facilitate the group if necessary. There were no gaps in the services of the
intervention to the youth. There are currently up to thirteen (13) youth in the two ongoing
Skillstreaming groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Skillstreaming the Adolescent. The monitoring was completed by the Director of Clinical Services
who is formally trained in the Skillstreaming curriculum. The program uses internal fidelity
monitoring checklists specific to Skillstreaming to document the monitoring. The program
completed checklists for all facilitators for each month they completed the groups except for the
month of 08/21.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Skillstreaming". Each tab had a copy of the fidelity checklist completed on the
facilitator(s) who perfomed groups during the specific month.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of Skillstreaming the Adolescent. The program uses internal fidelity
montoring/corrective action checklists specific to the Skillstreaming curriculum. Each completed
checklist includes feedback specifying either the facilitator did not require any corrective action or,
if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Clinical Services and a review of the documentation provided
verified the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received at
least a 90-day evaluation within the last year. The performance evaluationa for these staff
included entries of the staff's ability in delivering Skillstreaming the Adolescent. These included
comments on any strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 39 youth in the
sample, 100% (39 of 39) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 39 youth in the
sample, 95% (37 of 39) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 12 youth
High = 17 youth
Total Youth in Sample = 39 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 17 youth
Total Youth in Sample = 39 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (34 of 39) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 44% (17 of 39) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 81 85 95%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services.
39 youth completed the program during the review period. Of those youth, all 39 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 39 total youth sampled, all
39 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 39
received at least the recommended weeks of service. Youth in the sample completed between 22
and 56 weeks of service, with an average of 42 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 39 total youth
sampled, all 39 included dosage in the EBS Module. Of those youth with correct dosage, 37
received at least the recommended hours of service. Youth in the sample completed between 16
and 74 hours of service, with an average of 51 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Quality of Service Delivery score by ensuring facilitator turnover
does not impact groups and fidelity monitoring is completed every month for each facilitator.
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Amount of Service score by ensuring dosage for all youth is
recorded accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2021-22 Kissimmee MHOS-SAOS Skillstreaming.pdf

program_name_raw: Kissimmee Youth Academy

program_status_note: Open

service_type: Skillstreaming the Adolescent


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Borderline Developmental
Disability / Developmental Disability
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Impact of Crime
SPEP Review Period: 05/01/21-04/30/22
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Impact of Crime (IOC) intervention is being reviewed. The program maintains all SPEP
documentation in a tabbed binder separated into sections and organized by intervention. The
training protocol for this intervention is facilitated by either Department of Juvenile Justice certified
trainers or a Corporate certified Impact of Crime trainer. The training consists of reviewing and
learning the curriculum manual and conducting mock groups. During the review period from
05/01/21 to 04/30/22 there were seven staff formally trained in the IOC curriculum: one Director of
Case Management, two Transition Service Managers, and four Case Manager. One Director of
Case Management, three Case Managers and two Transition Service Managers facilitated group
during the review period. Three Case Managers left during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual Impact of Crime-Addressing the Harm. There is both a facilitator
and student manuals. The manual provides a script and instructions for the lessons. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered and includes detailed instructions for the delivery of each lesson. All youth receive the
Impact of Crime intervention as a part of their required programming for Restorative Justice.
Group times for Impact of Crime depend on dorm assignment and program schedule. During the
review period, IOC groups were held Mondays and Wednesdays from 4:30 to 5:30 or 5:30 to
6:30, Tuesdays and Thursdays from 5:30 to 6:30, and Wednesdays and Fridays from 5:30-6:30.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed via Microsoft Teams on 05/03/22 from 5:30 to 6:30
p.m. There were two staff and eight youth present in the group. The facilitator followed the lesson
plan according to the manual, did not read the lesson verbatim but rather used his own words to
convey the meaning of the discussed topic. He was very easy for the youth to follow. He ensured
the youth understood the concepts prior to moving forward in the lesson. The facilitator adhered to
the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Case
Management confirmed the program has had some turnover during the review period. The
program lost three Case Managers on 12/21/21, 09/08/21 and 02/26/22. None of the staff who left
were facilitating an Impact of Crime (IOC) group at the time of their departure. The turnover did
not adversely affect service delivery. The program's policy regarding anticipated turnover is
addressed by having administrative or other trained staff step in and facilitate the group if
necessary. There were no gaps in the services to the youth. There were between seven and nine
youth in the completed groups and there are seven, ten (10), and eight youth in the current
ongoing groups.
There were three cohorts of IOC completed during the review period. These groups ran 05/03/21-
09/07/21 (MHOS-BDD), 07/06/21-11/03/21 (MHOS-BDD) and 10/04/21-02/08/22 (SAOS-BDD)
There are three groups currently ongoing that both began on 12/01/22-(SAOS-MHOS-BDD),
01/11/22-SAOS-MHOS-BDD) and 04/25/22 (MHOS-BDD).
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Case Management who is formally trained in the Impact of Crime
(IOC) curriculum. The program uses internal fidelity monitoring checklists specific to IOC to
document the monitoring. The program had completed checklists for all facilitators for each month
they completed their IOC groups.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for IOC". Each tab had a copy of the fidelity checklist completed on the facilitator(s)
who perfomed groups during the specific month.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity montoring /
corrective action checklists specific to the Impact of Crime curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Case Management and a review of the documentation provided
verified the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the four current trained facilitators found these staff had
received an evaluation within the last year. The performance evaluations included entries of the
staff's skill in delivering Impact of Crime. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 21 youth in the
sample, 100% (21 of 21) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 21 youth in the
sample, 95% (20 of 21) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 2 youth
High = 12 youth
Total Youth in Sample = 21 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 21 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (18 of 21) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 57% (12 of 21) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 81 85 95%
This SPEP report evaluates Impact of Crime, an intervention delivered at Kissimmee Youth
Academy - Borderline Developmental Disability / Developmental Disability.
21 youth completed the program during the review period. Of those youth, all 21 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 21 total youth sampled, all
21 included dosage with end dates in the EBS Module. Of those youth, 21 received at least the
recommended weeks of service. Youth in the sample completed between 17 and 24 weeks of
service, with an average of 20 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 21 total youth
sampled, all 21 included dosage in the EBS Module. Of those youth, 20 received at least the
recommended hours of service. Youth in the sample completed between 22 and 32 hours of
service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
maintain their SPEP Quality of Service Delivery score by continuing the practices in place at the
time of this review.
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
optimize their SPEP Amount of Service score by ensuring dosage for all youth is recorded
accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Kissimmee BDD-DD IOC.pdf

program_name_raw: Kissimmee Youth Academy BDD and DD

service_type: Impact of Crime


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Borderline Developmental
Disability / Developmental Disability
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimmee, Florida 34744
Primary Service: Male Healthy Relationships
SPEP Review Period: 05/01/21-04/30/21
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Male Healthy Relationships bundle includes Young Men's Work and Teen Relationships. The
program maintains all Standardized Program Evaluation Protocol (SPEP) documentation in a
tabbed binder separated into sections and organized by intervention. There is a training protocol
for these interventions staff must attend, which facilitated by the Director of Clinical Services.
During the review period from 05/01/21 to 04/30/22 there were two licensed Therapists and seven
non-licensed Therapists trained in the curricula for Male Healthy Relationships. There were also
previous employees who facilitated the groups, two non-licensed Therapists and four licensed
Therapists. All Clinical staff are trained in these areas to manage facilitator turnover if it occurs, as
well as Regional and Corporate Clinical Directors who also provide coverage. The Director of
Clinical Services or her designee who is a licensed Therapist perform fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the brand name manuals "Young Men's Work" and "The Teen Relationships
Workbook" for the Male Healthy Relationships bundle. There are facilitator and student manuals.
The manual provides a script and instructions for lessons. The curriculum also includes
implementation guidelines, identifies the order the lessons should be delivered and includes
detailed instructions for the delivery of each lesson. All youth receive the Male Health
Relationships intervention as a part of their programming. Male Healthy Relationships groups are
open and ongoing groups held on Fridays from 2:00 to 3:00 and from 3:00 to 4:00 for mental
health overlay and the intensive youth. The substance abuse overlay youth receive the
intervention on Saturdays and Sundays between 8:30-11:30.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Male Healthy Relationships was observed on 05/13/22 from 2:30 to 3:30 p.m. A copy
of Session 22, Creating Relationships, Part 2, was provided to the reviewer prior to the group.
There were five youth and two staff members present for the session. The facilitator followed the
lesson plan according to the manual, did not read the lesson verbatim but rather used her own
words to convey the meaning of the discussed topic. She was very knowledgeable of the
materials. She was easy for the youth to follow and ensured the youth grasped the concepts prior
to moving forward in the lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation provided and an interview with the Director of Clinical Services
confirmed the program has had some turnover during the review period. The program lost two
non-licensed Therapists and four licensed Therapists; however, there were new Master Level
Therapists hired. This turnover did not adversely affect service delivery of Male Health
Relationships. The program's policy regarding anticipated turnover is addressed by having
Clinical Staff as well as Regional and Corporate Clinical staff trained to step in and facilitate the
groups if necessary. The program currently has two current ongoing groups. There were between
four to five youth in the Young Men's Work groups and between six and eight youth in the Teen
Relationships groups.
The program would also defer to the Director of Clinical Services should there be a gap longer
than one month following turnover if there is no one else to fill the gap. There was no disruption in
service of the intervention.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Clinical Services who is formally trained in the Male Healthy
Relationships bundle. The program uses internal fidelity monitoring checklists specific to Young
Men's Work and The Teen Relationships Workbook to document the monitoring. The program
had completed checklists monthly for all facilitators except for the months of 07/21, 08/21, 11/21
and 12/21.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Male Healthy Relationships". Each tab had a copy of the fidelity checklist
completed on the facilitator (s) who perfomed groups during the specific month. There are
currently two ongoing groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of Male Healthy Relationships. The program uses internal fidelity
montoring / corrective action checklists specific to the each curriculum. The completed checklist
includes feedback specifying either the facilitator did not require any corrective action or if the
facilitator did require corrective action and would include recommendations for future
improvement. The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Clinical Services and a review of the documentation provided
verified the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found that all Therapists received
at least a 90-day evaluation within the last year. The performance evaluations included entries of
the staff's skill in delivering the Male Healthy Relationships bundled services. These included
comments on any strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 21 youth in the
sample, 71% (15 of 21) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 21 youth in the
sample, 95% (20 of 21) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 2 youth
High = 12 youth
Total Youth in Sample = 21 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 21 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (18 of 21) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 57% (12 of 21) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 87 95 92%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Kissimmee
Youth Academy - Borderline Developmental Disability / Developmental Disability.
21 youth completed the program during the review period. Of those youth, all 21 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 21 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended weeks of service. Youth in the sample completed between 38
and 48 weeks of service, with an average of 42 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 21 total youth
sampled, all 21 included dosage in the EBS Module. Of those youth, 20 received at least the
recommended hours of service. Youth in the sample completed between 33 and 78 hours of
service, with an average of 73 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
optimize their SPEP Quality of Service Delivery score by ensuring fidelity monitoring is completed
every month for each facilitator.
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
optimize their SPEP Amount of Service score by ensuring dosage for all youth is recorded
accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Kissimmee BDD-DD MHR.pdf

service_type: Seeking Safety


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Borderline Developmental
Disability / Developmental Disability
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimmee, Florida 34744
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 05/01/21-04/30/22
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming intervention is being reviewed. There is a training protocol conducted by the
Director of Clinical Services or designee to include training in the Skillstreaming the Adolescent
currciulum, group observations, and debriefings. The program maintains all SPEP documentation
in a tabbed binder separated into sections and organized by intervention. During the review period
from 05/01/21 to 04/30/22 there were two licensed Therapists and seven non-licensed Therapists
trained in the curriculum. Seven non-licensed Therapists facilitated group during the review
period. The two licensed Therapists provided fidelity monitoring for the non-licensed Therapists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual in use is Skillstreaming the Adolescent, A Guide for Teaching Prosocial Skills, Third
Edition. There is a facilitator manual. The facilitator manual contains a script for the facilitator for
the groups. It provides lessons and includes implementation guidelines, identifies the order the
lessons should be delivered in and includes detailed instructions for the delivery of each lesson.
The intervention is held on Wednesdays from 2:30 to 3:30 and 3:30 to 4:30 and on Saturdays
from 11:30 to 12:30 and 4:30 to 5:30. The program maintains their records in SPEP binders which
are divided by intervention and BDD/DD, MHOS, or SAOS. All youth at Kissimmee Youth
Academy receive Skillstreaming. Skillstreaming is an open, ongoing group. There are up to
thirteen (13) youth in each ongoing group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
A session of Skillstreaming the Adolescent was observed on 05/03/22 from 11:25 to 12:25p.m.
There were two staff and nine youth present in the group. The facilitator provided a copy of the
the lesson "How to Communicate and Dealing with Contradicting Messages" to the reviewer. The
facilitator followed the lesson plan according to the manual, did not read the lesson verbatim but
rather used his own words to convey the meaning of the discussed topic. He was very
knowledgeable of the materials. He was easy for the youth to follow. He ensured the youth
understood the oncepts prior to moving forward in the lesson. The facilitator adhered to the lesson
plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Clinical
Services confirmed the program has had some turnover during the review period. The program
lost one Assistant Clinical Director and four Therapists during the review period. They have since
hired new staff and trained them. The Skillstreaming the Adolescent groups being facilitated by
the therapists who left were transitioned to other trained Therapists without a gap in service. The
program's policy regarding anticipated turnover is addressed by having administrative or other
trained staff step in and facilitate the group if necessary. There were no gaps in the services of the
intervention to the youth. There are currently up to thirteen (13) youth in the two ongoing
Skillstreaming groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Skillstreaming the Adolescent. The monitoring was completed by the Director of Clinical Services
who is formally trained in the Skillstreaming curriculum. The program uses internal fidelity
monitoring checklists specific to Skillstreaming to document the monitoring. The program
completed checklists for all facilitators for each month they completed the groups except for the
month of 08/21.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Skillstreaming". Each tab had a copy of the fidelity checklist completed on the
facilitator(s) who perfomed groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of Skillstreaming the Adolescent. The program uses internal fidelity
montoring/corrective action checklists specific to the Skillstreaming curriculum. Each completed
checklist includes feedback specifying either the facilitator did not require any corrective action or,
if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Clinical Services and a review of the documentation provided
verified the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received at
least a 90-day evaluation within the last year. The performance evaluationa for these staff
included entries of the staff's ability in delivering Skillstreaming the Adolescent. These included
comments on any strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 21 youth in the
sample, 71% (15 of 21) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 21 youth in the
sample, 100% (21 of 21) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 2 youth
High = 12 youth
Total Youth in Sample = 21 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 21 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (18 of 21) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 57% (12 of 21) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 79 85 93%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability.
21 youth completed the program during the review period. Of those youth, all 21 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 21 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended weeks of service. Youth in the sample completed between 38
and 48 weeks of service, with an average of 42 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 21 total youth
sampled, all 21 included dosage in the EBS Module. Of those youth with correct dosage 21
received at least the recommended hours of service. Youth in the sample completed between 36
and 50 hours of service, with an average of 49 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
optimize their SPEP Quality of Service Delivery score by ensuring facilitator turnover does not
impact groups and fidelity monitoring is completed every month for each facilitator.
Kissimmee Youth Academy - Borderline Developmental Disability / Developmental Disability can
optimize their SPEP Amount of Service score by ensuring dosage for all youth is recorded
accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Kissimmee BDD-DD Skillstreaming.pdf


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Mental Health Overlay Services /
Substance Abuse Overlay Services
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Impact of Crime
SPEP Review Period: 05/01/21-04/30/22
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Impact of Crime (IOC) intervention is being reviewed. The program maintains all SPEP
documentation in a tabbed binder separated into sections and organized by intervention. The
training protocol for this intervention is facilitated by either Department of Juvenile Justice certified
trainers or a Corporate certified Impact of Crime trainer. The training consists of reviewing and
learning the curriculum manual and conducting mock groups. During the review period from
05/01/21 to 04/30/22 there were seven staff formally trained in the IOC curriculum: one Director of
Case Management, two Transition Service Managers, and four Case Manager. One Director of
Case Management, three Case Managers and two Transition Service Managers facilitated group
during the review period. Three Case Managers left during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual Impact of Crime-Addressing the Harm. There is both a facilitator
and student manuals. The manual provides a script and instructions for the lessons. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered and includes detailed instructions for the delivery of each lesson. All youth receive the
Impact of Crime intervention as a part of their required programming for Restorative Justice.
Group times for Impact of Crime depend on dorm assignment and program schedule. During the
review period, IOC groups were held Mondays and Wednesdays from 4:30 to 5:30 or 5:30 to
6:30, Tuesdays and Thursdays from 5:30 to 6:30, and Wednesdays and Fridays from 5:30-6:30.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed via Microsoft Teams on 05/03/22 from 5:30 to 6:30
p.m. There were two staff and eight youth present in the group. The facilitator followed the lesson
plan according to the manual, did not read the lesson verbatim but rather used his own words to
convey the meaning of the discussed topic. He was very easy for the youth to follow. He ensured
the youth understood the concepts prior to moving forward in the lesson. The facilitator adhered to
the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Case
Management confirmed the program has had some turnover during the review period. The
program lost three Case Managers on 12/21/21, 09/08/21 and 02/26/22. None of the staff who left
were facilitating an Impact of Crime (IOC) group at the time of their departure. The turnover did
not adversely affect service delivery. The program's policy regarding anticipated turnover is
addressed by having administrative or other trained staff step in and facilitate the group if
necessary. There were no gaps in the services to the youth. There were between seven and nine
youth in the completed groups and there are seven, ten (10), and eight youth in the current
ongoing groups.
There were three cohorts of IOC completed during the review period. These groups ran 05/03/21-
09/07/21 (MHOS-BDD), 07/06/21-11/03/21 (MHOS-BDD) and 10/04/21-02/08/22 (SAOS-BDD)
There are three groups currently ongoing that both began on 12/01/22-(SAOS-MHOS-BDD),
01/11/22-SAOS-MHOS-BDD) and 04/25/22 (MHOS-BDD).
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Case Management who is formally trained in the Impact of Crime
(IOC) curriculum. The program uses internal fidelity monitoring checklists specific to IOC to
document the monitoring. The program had completed checklists for all facilitators for each month
they completed their IOC groups.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for IOC". Each tab had a copy of the fidelity checklist completed on the facilitator(s)
who perfomed groups during the specific month.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity montoring /
corrective action checklists specific to the Impact of Crime curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Case Management and a review of the documentation provided
verified the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the four current trained facilitators found these staff had
received an evaluation within the last year. The performance evaluations included entries of the
staff's skill in delivering Impact of Crime. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration e
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 39 youth in the
sample, 90% (35 of 39) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 39 youth in the
sample, 92% (36 of 39) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 12 youth
High = 17 youth
Total Youth in Sample = 39 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 17 youth
Total Youth in Sample = 39 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (34 of 39) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 44% (17 of 39) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 79 85 93%
This SPEP report evaluates Impact of Crime, an intervention delivered at Kissimmee Youth
Academy - Mental Health Overlay Services / Substance Abuse Overlay Services.
39 youth completed the program during the review period. Of those youth, all 39 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 39 total youth sampled, only
36 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 35
received at least the recommended weeks of service. Youth in the sample completed between 11
and 31 weeks of service, with an average of 19 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 39 total youth
sampled, only 37 included dosage in the EBS Module. Of those youth with correct dosage, 36
received at least the recommended hours of service. Youth in the sample completed between 22
and 50 hours of service, with an average of 33 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can maintain their SPEP Quality of Service Delivery score by continuing the practices in
place at the time of this review.
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Amount of Service score by ensuring dosage for all youth is
recorded accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Kissimmee MHOS-SAOS IOC.pdf