state: FL

facility_name: Kissimmee Youth Academy

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Truecore Behavioral
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Impact of Crime BDD-DD
SPEP Review Period: 05/01/23-04/30/24
SPEP Review Conducted: 05/14/24
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Impact of Crime (IOC) intervention is being reviewed for the dates of 05/01/23-04/30/24. The
program has a formal training protocol provided either by DJJ staff or corporate trainer. There
were seven staff trained during the review period, one director of case management, four case
managers and two transition service managers. Two case managers have resigned. All seven of
the staff facilitated Impact of Crime during the review period. The Director of Case Management
or her trained designee conducted fidelity monitoring. The program maintains all SPEP
documentation in a tabbed binder that is separated into sections which are then organized by
intervention. The training consists of reviewing and learning the curriculum manual and
conducting mock groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual "Impact of Crime-Addressing the Harm". There are both a facilitator
and student manuals. The manual provides a script and instructions for the lessons. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered and includes detailed instructions for the delivery of each lesson. All youth receive the
Impact of Crime intervention as a part of their required programming for Restorative Justice.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Impact of Crime group was observed via Zoom on Thursday 05/16/23 from 3:30pm-4:30pm.
There were two staff and eight youth present. The facilitator provided a copy of the the Lesson-
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Exercise Nine-"Peaceful Conflict Resolution" to the reviewer. The facilitator followed the lesson
plan according to the manual, did not read the lesson verbatim but rather used her own words to
convey the meaning of the discussed topic. She was very easy for the youth to follow. She
ensured that the youth understood the and grasped the concepts prior to moving forward in the
lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Case
Management confirmed that the program has had some turnover during the review period. Two
Case Managers resigned. The one Case Manager that was facilitating group left during the cohort
and the Director of Case Management took over the group and the cohort was completed so the
turnover did not adversly affect the services of Impact of Crime to the youth. The program's policy
regarding anticipated turnover is addressed by having administrative or other trained staff step in
and facilitate the group if necessary if there is no co-facilitator for that group. The groups also
overlap There was a short gap in the services of the Impact of Crime intervention to the youth
from 01/03/24-02/11/24-one month, eight days. This was due to the program waiting to have
enough new youth to start a new cohort which began on 02/12/24. There were seven cohorts of
IOC completed during the review period: 02/21/23-06/16/23, eight youth and one staff (SAOS),
03/02/23-07/07/23-nine youth and one staff (BDD/DD), 03/20/23-07/10/23 six youth and one staff
(MHOS), 06/15/23-11/14/23 eight youth and one staff (BDD/MHOS/MHOS MAX), 08/15/23-
11/30/23 five youth and one staff (SAOS/BDD/DD), 08/14/23-01/02/24 ten (10) youth and one
staff (MHOS/BDD/DD). There are also two ongoing groups that began on 02/12/24 with eight
youth and one staff (SAOS) and one that began on 02/20/24 with eight youth and one staff
(MHOS). Group times for Impact of Crime depend on dorm assignment and program schedule.
During the review period, IOC groups were held Mondays and Wednesdays from 4:30p-5:30p or
5:30p-6:30p and Tuesdays and Thursdays from 3:30p-4:30p.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Case Management or her designee who is formally trained in the
Impact of Crime curriculum. Seven staff facilitated the groups during the review period. The
program uses internal fidelity monitoring checklists specific to IOC to document the monitoring.
The program had completed checklists for all facilitators for each month they completed their IOC
groups. All completed fidelity checklists are maintained in the SPEP binder under a tab entitled
"Fidelity Monitoring for IOC". Each tab had a copy of the fidelity checklist completed on the
facilitator (s) who perfomed groups during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity
montoring/corrective action checklists specific to the Impact of Crime curriculum. Each completed
checklist includes feedback specifying either that the facilitator did not require any corrective
action or if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments that the previous
month's recommendations were applied and demonstrated. However, an interview with the
Director of Case Management and a review of the documentation provided verified that the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the five current trained facilitators found that all staff had
received either an annual or ninety day evaluation within the last year. The Performance
Evaluation for these staff found entries of the staff's ability in delivering Impact of Crime. These
included comments on any strengths or weaknesses of which the facilitator delivering the
curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 31 youth in the
sample, 45% (14 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 31 youth in the
sample, 55% (17 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 77%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 15 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 77% (24 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 48% (15 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 68 85 80%
This SPEP report evaluates Impact of Crime, an intervention delivered at Kissimmee Youth
Academy.
31 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 31 total youth sampled, only
21 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended weeks of service. Youth in the sample completed between 8
and 31 weeks of service, with an average of 17 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 21 included dosage in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended hours of service. Youth in the sample completed between 5
and 57 hours of service, with an average of 30 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Kissimmee Youth Academy can can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2023-24 Kissimmee Youth Academy BDD-DD IOC.pdf

program_name_raw: Kissimmee Youth Academy BDD and DD

program_status_note: Open

service_type: Impact of Crime


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Truecore Behavioral
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Seven Challenges-SAOS
SPEP Review Period: 05/01/23-04/30/24
SPEP Review Conducted: 05/14/24
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A technical assistance specialist reviewed the Seven Challenges intervention for the dates of
05/01/23-04/30/24. The program maintains all SPEP documentation in a tabbed binder organized
by intervention. The Seven Challenges group is held on Mondays, Tuesdays and Thursdays from
2:20p-3:20p for SAOS youth. The training protocol for this intervention is conducted by the Clinical
Director or designee to include review of the manual, group observation and debriefings. There
were two licensed therapists and eight non-licensed therapists trained in the Seven Challenges
curriculum. All eight of the non-licensed therapists facilitated group during the review period. The
two licensed therapists provided fidelity monitoring for the non-licensed therapists. Five of the
therapists left during the review period and five were hired and trained to replace them. There
were two cycles of Seven Challenges, one which is currently ongoing.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided to this reviewer was The Seven Challenges Manual, 2nd Edition by Robert
Schwebel, PhD. The facilitator manual provides detailed information about the curriculum and
some suggestions for other materials that can be used. There is not a specific script for this
curriculum.The manual lists core components for the curriculum such as readings, journaling,
education/counseling sessions as well as family sessions and provides extra ideas such as role
plays in the group setting. It also provides a set of group rules. There are separate manuals for
each lesson with objectives, materials and preparation. It also includes instructions for activities
and notes. It has scenarios for the youth and discussion points for the facilitator for the groups.
The group size is usually six to eight youth with a maximum of fifteen (15) youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Seven Challenges group was observed via Zoom on Tuesday 05/14/24 from 2:20p-3:20 p.m..
There were fourteen (14) youth and three staff present in the group. The facilitator provided a
copy of the the Lesson "Changing our Feelings and Forgetting Problems" to the reviewer. The
facilitator followed the lesson plan according to the manual, did not read the lesson verbatim but
rather used his own words to convey the meaning of the discussed topic. He was knowledgeable
of the materials. He was easy for the youth to follow. He also ensured that the youth understood
tthe concepts prior to moving forward. The facilitator adhered to the lesson plan.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's documentation and an interview with the clinical director confirmed that the
program has had some turnover during the review period. The program lost five therapists.
However, the turnover did not adversly affect the services of Seven Challenges to the youth. The
other trained therapists continued facilitating the groups. The program hired and trained five new
therapists.
The program's policy regarding anticipated turnover is addressed by having administrative or
other trained staff step in and facilitate the group if necessary. There were no gaps in the services
of the intervention to the youth.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program performs internal fidelity monitoring. The monitoring was completed by the clinical
director who is formally trained in the Seven Challenges curriculum. The program uses internal
fidelity monitoring checklists specific to Seven Challenges to document the monitoring. The
program had completed checklists for all facilitators for each month they completed the groups.
There was no missed monitoring during the review period. All completed fidelity checklists are
maintained in the SPEP binder under a tab entitled "Fidelity Monitoring for Seven Challenges".
Each tab had a copy of the fidelity checklist completed on the facilitator (s) who perfomed groups
during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective action based on the fidelity/monitoring. The program
uses internal fidelity montoring/corrective action checklists specific to the Seven Challenges
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
curriculum. Each completed checklist includes feedback specifying either that the facilitator did
not require any corrective action or if the facilitator did require corrective action, it would include
recommendations for future improvement. The following month's checklist would then include
comments that the previous month's recommendations were applied and demonstrated. However,
an interview with the clinical director and a review of the documentation provided verified that the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Employee records for the facilitators found that all therapists except for three received annual
evaluations within the last year. The other three received ninety (90) day evaluations.
The program can earn 1 point if there is evidence of annual performance evaluations of facilitators
including assessment of the facilitator's proficiency in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 74% (14 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 95% (18 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 79%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 37%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 79% (15 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 37% (7 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 84 95 88%
This SPEP report evaluates Seven Challenges, an intervention delivered at Kissimmee Youth
Academy.
73 youth completed the program during the review period. Of those youth, 19 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 19 total youth sampled, only
14 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended weeks of service. Youth in the sample completed between 28
and 53 weeks of service, with an average of 39 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, only 19 included dosage in the EBS Module. Of those youth with correct dosage, 18
received at least the recommended hours of service. Youth in the sample completed between 24
and 280 hours of service, with an average of 81 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Kissimmee Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Kissimmee Youth Academy 7 Challenges.pdf

program_name_raw: Kissimmee Youth Academy

service_type: Seeking Safety


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Truecore Behavioral
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Skillstreaming-BDD/DD
SPEP Review Period: 05/01/23-04/30/24
SPEP Review Conducted: 05/14/24
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming intervention is being reviewed for the dates of 05/01/23-04/30/24. There is a
formal training protocol which is completed by the Clinical Director or his designee to include
training in the Skillstreaming currciulum, group observations and debriefings. During the review
period, there were two cohorts of Skillstreaming, one which is currently ongoing with six youth in
the group. The program maintains all SPEP documentation in a tabbed binder that is separated
into sections which are then organized by intervention. The Skillstreaming groups are held on
Tuesdsays and Thursdays from 2:20p-3:20p and 3:20p-4:20p. During the review period there
were two licensed Therapists and eight non-licensed Therapists trained in the curriculum. The
non-licensed Therapists facilitated group during the review period.The two licensed Therapists
provided fidelity monitoring for the non-licensed Therapists. Five Therapists resigned during the
review period.and five were hired and trained to replace them. The remaining Therapists
continued facilitating the groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided to this reviewer was Skillstreaming the Adolescent, A Guide for Teaching
Prosocial Skills, Third Edition. There is a facilitator manual. The facilitator manual contains a script
for the facilitator for the groups. It provides lessons and includes implementation guidelines,
identifies the order the lessons should be delivered in and includes detailed instructions for the
delivery of each lesson. The intervention is held on Tuesdays and Thursdays from 2:20p-3:20p,
3:20p-4:20p, and 11:08a-12:08p. The program maintains their records in SPEP binders which are
divided by intervention and designation of youth as to BDD/DD, MHOS, or SAOS. All youth at
Kissimmee Youth Academy receive Skillstreaming. Skillstreaming is an open, ongoing group.
At the time of the review, the program met the criteria to earn a score of 2.
c. Observed Adherence to the Manual/Protocol Basic Score: 1 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Skillstreaming group was observed in person on Tuesday 05/14/24 from 3:20p-4:20p virtually
via Zoom by S. Wilson, TAS. There were three staff and six youth present in the group. The
facilitator provided a copy of the the Lesson 25-"Negotiating" to the reviewer prior to the group.
The facilitator followed the lesson plan according to the manual, did not read the lesson verbatim
but rather used her own words to convey the meaning of the discussed topic. She was very
knowledgeable of the materials. She was very easy for the youth to follow. She ensured that the
youth understood and grasped each concept prior to moving forward in the lesson. The facilitator
adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Clinical Director
confirmed that the program has had turnover during the review period. The program lost five
Therapists during the review period.The turnover did not adversly affect the services of
Skillstreaming to the youth. The other trained Therapists continued facilitating the groups. New
Therapists have since been hired and trained. The program's policy regarding anticipated
turnover is addressed by having administrative or other trained staff step in and facilitate the
group if necessary. There were no gaps in the services of the intervention to the youth.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Clinical Director who is formally trained in the Skillstreaming curriculum and/or
the designee. The program uses internal fidelity monitoring checklists specific to Skillstreaming to
document the monitoring. The program had completed checklists for all facilitators for each month
they completed the groups except for the months of December 2023 and January 2023. All
completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Skillstreaming". Each tab had a copy of the fidelity checklist completed on the
facilitator (s) who perfomed groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity
montoring/corrective action checklists specific to the Skillstreaming curriculum. Each completed
checklist includes feedback specifying either that the facilitator did not require any corrective
action or if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments that the previous
month's recommendations were applied and demonstrated. However, an interview with the
Clinical Director and a review of the documentation provided verified that the program did not take
any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found that these staff had
received at least either a 90 day evaluation or an annual review within the last year. The
Performance Evaluation for these staff found entries of the staff's ability in delivering the
curriculum. These included comments on any strengths or weaknesses of which the facilitator
delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 31 youth in the
sample, 58% (18 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 31 youth in the
sample, 100% (31 of 31) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 77%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 15 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 77% (24 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 48% (15 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 74 85 87%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy.
31 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 31 total youth sampled, only
18 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 18
received at least the recommended weeks of service. Youth in the sample completed between 27
and 79 weeks of service, with an average of 40 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 31 included dosage in the EBS Module. Of those youth with correct dosage, 31
received at least the recommended hours of service. Youth in the sample completed between 28
and 350 hours of service, with an average of 104 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Kissimmee Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Kissimmee Youth Academy BDD-DD Skillstreaming.pdf

service_type: Skillstreaming the Adolescent


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Truecore Behavioral
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Skillstreaming MHOS & SAOS
SPEP Review Period: 05/01/23-04/30/24
SPEP Review Conducted: 05/14/24
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming intervention is being reviewed for the dates of 05/01/23-04/30/24. There is a
formal training protocol which is completed by the Clinical Director or his designee to include
training in the Skillstreaming currciulum, group observations and debriefings. During the review
period, there were two cohorts of Skillstreaming, one which is currently ongoing with six youth in
the group. The program maintains all SPEP documentation in a tabbed binder that is separated
into sections which are then organized by intervention. The Skillstreaming groups are held on
Tuesdsays and Thursdays from 2:20p-3:20p and 3:20p-4:20p. During the review period there
were two licensed Therapists and eight non-licensed Therapists trained in the curriculum. The
non-licensed Therapists facilitated group during the review period.The two licensed Therapists
provided fidelity monitoring for the non-licensed Therapists. Five Therapists resigned during the
review period.and five were hired and trained to replace them. The remaining Therapists
continued facilitating the groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided to this reviewer was Skillstreaming the Adolescent, A Guide for Teaching
Prosocial Skills, Third Edition. There is a facilitator manual. The facilitator manual contains a script
for the facilitator for the groups. It provides lessons and includes implementation guidelines,
identifies the order the lessons should be delivered in and includes detailed instructions for the
delivery of each lesson. The intervention is held on Tuesdays and Thursdays from 2:20p-3:20p,
3:20p-4:20p, and 11:08a-12:08p. The program maintains their records in SPEP binders which are
divided by intervention and designation of youth as to BDD/DD, MHOS, or SAOS. All youth at
Kissimmee Youth Academy receive Skillstreaming. Skillstreaming is an open, ongoing group.
At the time of the review, the program met the criteria to earn a score of 2.
c. Observed Adherence to the Manual/Protocol Basic Score: 1 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Skillstreaming group was observed in person on Tuesday 05/14/24 from 3:20p-4:20p virtually
via Zoom by S. Wilson, TAS. There were three staff and six youth present in the group. The
facilitator provided a copy of the the Lesson 25-"Negotiating" to the reviewer prior to the group.
The facilitator followed the lesson plan according to the manual, did not read the lesson verbatim
but rather used her own words to convey the meaning of the discussed topic. She was very
knowledgeable of the materials. She was very easy for the youth to follow. She ensured that the
youth understood and grasped each concept prior to moving forward in the lesson. The facilitator
adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Clinical Director
confirmed that the program has had turnover during the review period. The program lost five
Therapists during the review period.The turnover did not adversly affect the services of
Skillstreaming to the youth. The other trained Therapists continued facilitating the groups. New
Therapists have since been hired and trained. The program's policy regarding anticipated
turnover is addressed by having administrative or other trained staff step in and facilitate the
group if necessary. There were no gaps in the services of the intervention to the youth.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Clinical Director who is formally trained in the Skillstreaming curriculum and/or
the designee. The program uses internal fidelity monitoring checklists specific to Skillstreaming to
document the monitoring. The program had completed checklists for all facilitators for each month
they completed the groups except for the months of December 2023 and January 2023. All
completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Skillstreaming". Each tab had a copy of the fidelity checklist completed on the
facilitator (s) who perfomed groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity
montoring/corrective action checklists specific to the Skillstreaming curriculum. Each completed
checklist includes feedback specifying either that the facilitator did not require any corrective
action or if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments that the previous
month's recommendations were applied and demonstrated. However, an interview with the
Clinical Director and a review of the documentation provided verified that the program did not take
any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found that these staff had
received at least either a 90 day evaluation or an annual review within the last year. The
Performance Evaluation for these staff found entries of the staff's ability in delivering the
curriculum. These included comments on any strengths or weaknesses of which the facilitator
delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 67 youth in the
sample, 48% (32 of 67) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 67 youth in the
sample, 93% (62 of 67) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 73%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 6 youth
Moderate-High = 17 youth
High = 26 youth
Total Youth in Sample = 67 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 26 youth
Total Youth in Sample = 67 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 73% (49 of 67) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 39% (26 of 67) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 18 25 72%
Totals 70 85 82%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy.
73 youth completed the program during the review period. Of those youth, 67 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 67 total youth sampled, only
34 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 32
received at least the recommended weeks of service. Youth in the sample completed between 1
and 65 weeks of service, with an average of 34 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 67 total youth
sampled, only 62 included dosage in the EBS Module. Of those youth with correct dosage, 62
received at least the recommended hours of service. Youth in the sample completed between 40
and 241 hours of service, with an average of 89 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Kissimmee Youth Academy Skillstreaming.pdf


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Truecore Behavioral
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Impact of Crime MHOS-SAOS
SPEP Review Period: 05/01/23-04/30/24
SPEP Review Conducted: 05/14/24
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Impact of Crime (IOC) intervention is being reviewed for the dates of 05/01/23-04/30/24. The
program has a formal training protocol provided either by DJJ staff or corporate trainer. There
were seven staff trained during the review period, one director of case management, four case
managers and two transition service managers. Two case managers have resigned. All seven of
the staff facilitated Impact of Crime during the review period. The Director of Case Management
or her trained designee conducted fidelity monitoring. The program maintains all SPEP
documentation in a tabbed binder that is separated into sections which are then organized by
intervention. The training consists of reviewing and learning the curriculum manual and
conducting mock groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual "Impact of Crime-Addressing the Harm". There are both a facilitator
and student manuals. The manual provides a script and instructions for the lessons. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered and includes detailed instructions for the delivery of each lesson. All youth receive the
Impact of Crime intervention as a part of their required programming for Restorative Justice.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Impact of Crime group was observed via Zoom on Thursday 05/16/23 from 3:30pm-4:30pm.
There were two staff and eight youth present. The facilitator provided a copy of the the Lesson-
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Exercise Nine-"Peaceful Conflict Resolution" to the reviewer. The facilitator followed the lesson
plan according to the manual, did not read the lesson verbatim but rather used her own words to
convey the meaning of the discussed topic. She was very easy for the youth to follow. She
ensured that the youth understood the and grasped the concepts prior to moving forward in the
lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Case
Management confirmed that the program has had some turnover during the review period. Two
Case Managers resigned. The one Case Manager that was facilitating group left during the cohort
and the Director of Case Management took over the group and the cohort was completed so the
turnover did not adversly affect the services of Impact of Crime to the youth. The program's policy
regarding anticipated turnover is addressed by having administrative or other trained staff step in
and facilitate the group if necessary if there is no co-facilitator for that group. The groups also
overlap There was a short gap in the services of the Impact of Crime intervention to the youth
from 01/03/24-02/11/24-one month, eight days. This was due to the program waiting to have
enough new youth to start a new cohort which began on 02/12/24. There were seven cohorts of
IOC completed during the review period: 02/21/23-06/16/23, eight youth and one staff (SAOS),
03/02/23-07/07/23-nine youth and one staff (BDD/DD), 03/20/23-07/10/23 six youth and one staff
(MHOS), 06/15/23-11/14/23 eight youth and one staff (BDD/MHOS/MHOS MAX), 08/15/23-
11/30/23 five youth and one staff (SAOS/BDD/DD), 08/14/23-01/02/24 ten (10) youth and one
staff (MHOS/BDD/DD). There are also two ongoing groups that began on 02/12/24 with eight
youth and one staff (SAOS) and one that began on 02/20/24 with eight youth and one staff
(MHOS). Group times for Impact of Crime depend on dorm assignment and program schedule.
During the review period, IOC groups were held Mondays and Wednesdays from 4:30p-5:30p or
5:30p-6:30p and Tuesdays and Thursdays from 3:30p-4:30p.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Case Management or her designee who is formally trained in the
Impact of Crime curriculum. Seven staff facilitated the groups during the review period. The
program uses internal fidelity monitoring checklists specific to IOC to document the monitoring.
The program had completed checklists for all facilitators for each month they completed their IOC
groups. All completed fidelity checklists are maintained in the SPEP binder under a tab entitled
"Fidelity Monitoring for IOC". Each tab had a copy of the fidelity checklist completed on the
facilitator (s) who perfomed groups during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity
montoring/corrective action checklists specific to the Impact of Crime curriculum. Each completed
checklist includes feedback specifying either that the facilitator did not require any corrective
action or if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments that the previous
month's recommendations were applied and demonstrated. However, an interview with the
Director of Case Management and a review of the documentation provided verified that the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the five current trained facilitators found that all staff had
received either an annual or ninety day evaluation within the last year. The Performance
Evaluation for these staff found entries of the staff's ability in delivering Impact of Crime. These
included comments on any strengths or weaknesses of which the facilitator delivering the
curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 67 youth in the
sample, 60% (40 of 67) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 67 youth in the
sample, 58% (39 of 67) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 73%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 6 youth
Moderate-High = 17 youth
High = 26 youth
Total Youth in Sample = 67 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 26 youth
Total Youth in Sample = 67 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 73% (49 of 67) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 39% (26 of 67) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 18 25 72%
Totals 68 85 80%
This SPEP report evaluates Impact of Crime, an intervention delivered at Kissimmee Youth
Academy.
73 youth completed the program during the review period. Of those youth, 67 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 67 total youth sampled, only
49 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 40
received at least the recommended weeks of service. Youth in the sample completed between 7
and 23 weeks of service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 67 total youth
sampled, only 49 included dosage in the EBS Module. Of those youth with correct dosage, 39
received at least the recommended hours of service. Youth in the sample completed between 2
and 32 hours of service, with an average of 28 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy can can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Kissimmee Youth Academy MHOS-SAOS IOC.pdf