summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - Mental Health Overlay Services /
Substance Abuse Overlay Services
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Seven Challenges
SPEP Review Period: 05/01/21-04/30/22
SPEP Review Conducted: 05/03/22
Florida Department of Juvenile Justice
Report Date: 8/31/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The 7 Challenges intervention is being reviewed. The program maintains all Standardized
Program Evaluation Protocol documentation in a tabbed binder separated into sections which are
then organized by intervention. The training protocol for this intervention is conducted by the
Clinical Director or designee to include training of the curriculum, review of the manual, group
observation and debriefings. During the review period from 05/01/21 to 04/30/22 there were two
licensed Therapists and seven non-licensed Therapists who were trained and facilitated groups
during the review period. The two licensed Therapists provided fidelity monitoring for the non-
licensed Therapists. Three of the Therapists left during the review period. The remaining
Therapists continued facilitating this open group.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided for review was The Seven Challenges Manual, 2nd Edition by Robert
Schwebel, PhD. The facilitator manual provides detailed information about the curriculum and
some suggestions for other materials that can be used. There is not a specific script for this
curriculum. The manual lists core components for the curriculum such as readings, journaling,
education/counseling sessions as well as family sessions and provides extra ideas such as role
plays in the group setting. It also provides a set of group rules. There are separate manuals for
each lesson with objectives, materials and preparation. It also includes instructions for activities
and notes. It has scenarios for the youth and discussion points for the facilitator for the groups.
The group size is five with a maximum of fifteen (15). The Seven Challenges group is held on
Sundays and Mondays from 11:25-12:25 p.m.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges group was observed via Zoom on 05/05/22 from 11:25 to 12:25
p.m. There were two staff and thirteen (13) youth present in the group. The group was working on
the lesson "We Have Been Called Names". The facilitator followed the lesson plan according to
the manual, did not read the lesson verbatim but rather used her own words to convey the
meaning of the discussed topic. She was very knowledgeable of the materials. She was easy for
the youth to follow. She ensured the youth grasped the concepts prior to moving forward in the
lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Clinical
Services confirmed the program has had some turnover during the review period. The program
lost one Assistant Clinical Director and four Therapists during the review period. They have hired
new staff since that time and trained them. Other trained Therapists continued facilitating the
groups being facilitated by the staff who left. The program's policy regarding anticipated turnover
is addressed by having administrative or other trained staff step in and facilitate the group if
necessary. There were no gaps in the services of the intervention to the youth. There are
currently thirteen (13) youth in the ongoing two groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring. The monitoring was
completed by the Director of Clinical Services who is formally trained in the Seven Challenges
curriculum. The program uses internal fidelity monitoring checklists specific to Seven Challenges
to document the monitoring. The program had completed checklists for all facilitators for each
month they completed the groups except for the months of 08/21, 01/22, 02/22 and 03/22. Fidelity
monitoring was missed due to staffing issues.
All completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for 7 Challenges". Each tab had a copy of the fidelity checklist completed on the
facilitator(s) who conducted groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of the primary service. The program uses internal fidelity montoring /
corrective action checklists specific to the Seven Challenges curriculum. Each completed
checklist includes feedback specifying either the facilitator did not require any corrective action or
if the facilitator did require corrective action, it would include recommendations for future
improvement. The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the Director of Clinical Services and a review of the documentation provided
verified that the program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received at
least a 90-day evaluation within the last year. The performance evaluations included entries of the
staff's skill in delivering Seven Challenges. These included comments on any strengths or
weaknesses of which the facilitator delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 89% (17 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 100% (19 of 19) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (17 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 91 95 96%
This SPEP report evaluates Seven Challenges, an intervention delivered at Kissimmee Youth
Academy - Mental Health Overlay Services / Substance Abuse Overlay Services.
39 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 19 total youth sampled, only
17 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended weeks of service. Youth in the sample completed between 38
and 56 weeks of service, with an average of 42 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, all 19 included dosage in the EBS Module. Of those youth, 19 received at least the
recommended hours of service. Youth in the sample completed 70 hours of service.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Quality of Service Delivery score by ensuring facilitator turnover
does not impact groups and fidelity monitoring is completed every month for each facilitator.
Kissimmee Youth Academy - Mental Health Overlay Services / Substance Abuse Overlay
Services can optimize their SPEP Amount of Service score by ensuring dosage for all youth is
recorded accurately in EBS and by ensuring youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Kissimmee MHOS-SAOS 7C.pdf

service_type: Seven Challenges


state: FL

facility_name: Kissimmee Youth Academy

summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 05/01/22-04/30/23
SPEP Review Conducted: 05/09/23
Florida Department of Juvenile Justice
Report Date: 10/24/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming the Adolescent intervention is being reviewed. There is a formal training
protocol which is completed by the clinical director or his designee to include training in the
Skillstreaming currciulum, group observations, and debriefings. During the review period from
05/01/2022 through 04/30/2023, there were three licensed therapists and twelve (12) non-
licensed therapists trained in the curriculum. The twelve (12) non-licensed therapists facilitated
group during the review period. The three licensed therapists provided fidelity monitoring for the
non-licensed therapists. The prior clinical director, one licensed therapist and five of the non-
licensed therapists left during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided for review was Skillstreaming the Adolescent, A Guide for Teaching
Prosocial Skills, Third Edition. There is a facilitator manual which contains a script for the
facilitator for the groups. It provides lessons and includes implementation guidelines, identifies the
order the lessons should be delivered in, and includes detailed instructions for the delivery of each
lesson.
The intervention is held on Tuesdays and Thursdays from 2:20-3:20, 3:20-4:20, and 11:08-12:08.
The program maintains their records in SPEP binders which are divided by intervention and
designation of youth as to BDD/DD, MHOS, or SAOS. All youth at Kissimmee Youth Academy
receive Skillstreaming. Skillstreaming is an open, ongoing group.
The Skillstreaming groups are held as follows: East Wing-There are two groups held on Tuesdays
and Thursdays for SAOS from 2:20-3:20 and 3:20-4:20.
There are two groups for BDD held on Tuesdays and Thursdays from 2:20-3:20 and 3:20-4:20.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Max groups are held on Tuesdays and Thursday from 2:20-3:20 and 3:20-4:20.
For West Wing, Skillstreaming is held for MHOS on Tuesdsays and Thursdays from 2:20-3:20
and from 3:20-4:20.
The BDD groups are held on Tuesdays and Thursday from 2:20-3:20.
Two groups for MAX are held on Tuesdays and Thursdays from 11:08-12:08.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed on 05/09/2023 from 11:08 to 12:08.
There were two staff and eight youth present in the group. The facilitator provided a copy of the
lesson, Expressing Affection, to the reviewer. The facilitator followed the lesson plan according to
the manual, did not read the lesson verbatim but rather used her own words to convey the
meaning of the discussed topic. She was knowledgeable of the materials and was easy for the
youth to follow. She ensured the youth understood each concept prior to moving forward in the
lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation provided and an interview with the clinical director confirmed the
program has had some Skillstreaming the Adolescent facilitator turnover during the review period.
The program lost the prior clinical director, one licensed therapist and five therapists during the
review period. The other trained therapists continued facilitating the groups. The program's policy
regarding anticipated turnover is addressed by having administrative or other trained staff step in
to facilitate the group if necessary. There were no gaps in the services of the intervention to the
youth. There are currently up to nine youth in the two ongoing groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Skillstreaming the Adolescent. The monitoring was completed by the clinical director who is
formally trained in the curriculum, or a trained designee. The program uses internal fidelity
monitoring checklists specific to Skillstreaming to document the monitoring. The program had
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
completed checklists for all facilitators for each month they completed the groups except for the
month of 06/22 and 03/23. All completed fidelity checklists are maintained in the SPEP binder
under a tab entitled "Fidelity Monitoring for Skillstreaming". Each tab had a copy of the fidelity
checklist completed on the facilitators who facilitated groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of Skillstreaming the Adolescent. The program uses internal fidelity
montoring/corrective action checklists specific to the curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or, if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the clinical director and a review of the documentation provided verified the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received at
least either a ninety (90)-day evaluation or an annual review within the last year. The performance
evaluationsfound entries of the staff's skill in delivering Skillstreaming the Adolescent. These
included comments on any strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 44 youth in the
sample, 80% (35 of 44) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 44 youth in the
sample, 95% (42 of 44) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 77%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 11 youth
High = 18 youth
Total Youth in Sample = 44 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 18 youth
Total Youth in Sample = 44 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 77% (34 of 44) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 41% (18 of 44) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 76 85 89%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy.
45 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 44 total youth sampled, only
35 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 35
received at least the recommended weeks of service. Youth in the sample completed between 18
and 60 weeks of service, with an average of 42 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 44 total youth
sampled, all 44 included dosage in the EBS Module. Of those youth, 42 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 10 and 424 hours of
service, with an average of 151 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 95%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups and fidelity monitoring is completed every
month for each facilitator.
Kissimmee Youth Academy can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2022-23 KYA MHOS-SAOS Skillstreaming.pdf

program_name_raw: Kissimmee Youth Academy

program_status_note: Open

service_type: Skillstreaming the Adolescent


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - BDD/DD
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Impact of Crime
SPEP Review Period: 05/01/22-04/30/23
SPEP Review Conducted: 05/09/23
Florida Department of Juvenile Justice
Report Date: 10/24/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Impact of Crime (IOC) intervention is being reviewed for the dates of 05/01/2022 through
04/30/2023. The program has a formal training protocol provided either by department staff or a
corporate trainer formally trained in Impact of Crime curriculum. There were five staff trained
during the review period: one Director of Case Management, two case managers, and two
Transition Service Managers. One Case Manager resigned, and one transition service manager
was promoted to Human Resources. All five staff facilitated Impact of Crime during the review
period. The director of case management conducted fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual "Impact of Crime-Addressing the Harm". There are both facilitator
and student manuals. The manual provides a script and instructions for the lessons. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered and includes detailed instructions for the delivery of each lesson. All youth receive the
Impact of Crime (IOC) intervention as a part of their required programming for Restorative Justice.
The program maintains all SPEP documentation in a tabbed binder separated into sections which
are then organized by intervention. Group times for Impact of Crime depend on dorm assignment
and program schedule. During the review period, IOC groups were held Mondays and
Wednesdays from 4:30 to 5:30 or 5:30 to 6:30, Tuesdays and Thursdays from 5:30 to 6:30, and
Wednesdays and Fridays from 5:30-6:30.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed via Zoom on Monday, 05/15/2023 from 4:30p.m. to
5:30p.m. There were two staff and six youth present in the group. The facilitator provided a copy
of the the Lesson, Chapter 5 "Ripple Effect of Crime; The Impact of Poor Decisions" to the
reviewer. The facilitator followed the lesson plan according to the manual, did not read the lesson
verbatim but rather used her own words to convey the meaning of the discussed topic. She was
very easy for the youth to follow. She ensured the youth understood the concepts prior to moving
forward in the lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation provided and an interview with the director of case management
confirmed the program has had some turnover during the review period. One case manager
resigned and one transition services manager was promoted to human resources. The case
manager who was facilitating group left during the cohort and the co-facilitator took over the
group. The transition services manager was not facilitating group at the time of their departure.
The turnover did not adversely affect the services of Impact of Crime (IOC) to the youth. The
program's policy regarding anticipated turnover is addressed by having administrative or other
trained staff step in and facilitate the group if necessary if there is no co-facilitator for that group.
There were no gaps in the services of the Impact of Crime intervention to the youth.
There were six cohorts of IOC completed during the review period, with three ongoing at the time
of the annual review. There were between seven and nine youth in the completed groups. The
first cohort ran 04/25/2022 through 08/25/2022 (MHOS, BDD/DD). The second cohort ran
06/07/2022 through10/22/2022 (MHOS, SAOS, BDD/DD). The third cohort ran 08/22/2022
through 12/19/2022 (BDD/DD, MHOS). The fourth cohort ran 10/03/2022 through 01/26/2023
(MHOS, BDD/DD, SAOS). The fifth cohort ran 10/03/2022 through 02/21/2023 (MHOS MAX). The
sixth cohort started 02/21/2023 and was ongoing at the time of the annual review with eleven (11)
BDD youth and two facilitators. The seventh cohort started 02/03/2023 and was ongoing at the
time of the annual review with seven SAOS youth and two facilitators. The eighth cohort started
03/02/2023 and was ongoing at the time of the annual review with seven BDD/DD youth and two
facilitators. The ninth cohort started 03/20/2003 and was ongoing at the time of the annual review
with eight MHOS youth and two facilitators.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Impact of Crime (IOC). The monitoring was completed by the director of case management, who
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
is formally trained in the curriculum. The program uses internal fidelity monitoring checklists
specific to IOC to document the monitoring. The program had completed checklists for all
facilitators for each month they completed IOC groups. All completed fidelity checklists are
maintained in the SPEP binder under a tab entitled "Fidelity Monitoring for IOC". Each tab had a
copy of the fidelity checklist completed on the facilitators who facilitated groups during the specific
month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of Impact of Crime (IOC). The program uses internal fidelity
montoring/corrective action checklists specific to the curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or, if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the director of case management and a review of the documentation provided
verified the program did not take any corrective actions during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received an
evaluation within the last year. The performance evaluations for these staff found entries of the
staff's skill in delivering Impact of Crime. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 35 youth in the
sample, 46% (16 of 35) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 35 youth in the
sample, 51% (18 of 35) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 8 youth
High = 17 youth
Total Youth in Sample = 35 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 17 youth
Total Youth in Sample = 35 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (29 of 35) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 49% (17 of 35) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 68 85 80%
This SPEP report evaluates Impact of Crime, an intervention delivered at Kissimmee Youth
Academy - BDD/DD.
36 youth completed the program during the review period. Of those youth, 35 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 35 total youth sampled, only
31 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 5
and 31 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 35 total youth
sampled, only 31 included dosage in the EBS Module. Of those youth with correct dosage, 18
received at least the recommended hours of service. Youth in the sample completed between 4
and 57 hours of service, with an average of 25 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy - BDD/DD can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Kissimmee Youth Academy - BDD/DD can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 KYA BDD-DD IOC.pdf

program_name_raw: Kissimmee Youth Academy BDD and DD

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - BDD/DD
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimmee, Florida 34744
Primary Service: Male Healthy Relationships
SPEP Review Period: 05/01/22-04/30/23
SPEP Review Conducted: 05/09/23
Florida Department of Juvenile Justice
Report Date:
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Male Healthy Relationships intervention is being reviewed for the dates of 05/01/2022
through 04/30/2023. This bundle includes the interventions Young Men's Work and The Teen
Relationships Workbook. The program has a formal training protocol for both groups which is
provided by the clinical director and includes review of the manual, group observations and
debriefings. During the review period, there were three licensed therapists and twelve (12) non-
licensed therapists trained in the curricula for Male Healthy Relationships.The twelve (12) non-
licensed therapists facilitated the groups. The clinical director or his designee who is a licensed
therapist perform fidelity monitoring for the non-licensed therapists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the brand-name manuals "Young Men's Work" and "The Teen Relationships
Workbook" for the Male Healthy Relationships bundled intervention. There are facilitator and
student manuals. The manual provides a script and instructions for lessons. The curriculum also
includes the lessons, implementation guidelines, identifies the order the lessons should be
delivered, and includes detailed instructions for the delivery of each session. The program
maintains all SPEP documentation in a tabbed binder separated into sections and then organized
by intervention.
All youth receive the Male Health Relationships intervention as a part of their programming. Male
Healthy Relationships groups are open and ongoing. The Male Healthy Relationships groups are
held on Wednesdays and Sundays.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Male Healthy Relationships, specifically Young Men's Work, was observed via Zoom
on Wednesday, 05/10/2023, from 2:30p.m. to 3:30p.m. A copy of the lesson, Becoming Allies,
was provided for review prior to the group. There were seven youth and two staff members
present for the session.The facilitator followed the lesson plan according to the manual. She did
not read the lesson verbatim but rather used her own words to convey the meaning of the
discussed topic. She was very knowledgeable of the materials. She was easy for the youth to
follow and ensured the youth understood the concepts prior to moving forward in the lesson. The
facilitator adhered to the lesson plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the clinical director
services confirmed the program has had some turnover during the review period. The program
lost the prior clinical director, one licensed therapist, and four non-licensed therapists during the
review period. A new clinical director has been appointed. The non-licensed therapists were
facilitating groups at the time of their departure. Those groups were transitioned to new facilitators
without a gap in service. The program's policy regarding anticipated turnover is addressed by
having clinical staff, as well as regional and corporate clinical staff, trained to step in and facilitate
the groups if necessary. The program currently has two ongoing groups of Young Men's Work
and Teen Relationships. There were between six to eight youth in each group.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of Male
Healthy Relationships. The monitoring was completed by the clinical director who is formally
trained in the service. The program uses internal fidelity monitoring checklists specific to Young
Men's Work and Teen Relationships to document the monitoring. The program completed
checklists monthly for all facilitators except for the months of 05/22, 10/22, 12/22, 02/23, 03/23. All
completed fidelity checklists are maintained in the SPEP binder under a tab entitled "Fidelity
Monitoring for Male Healthy Relationships". Each tab had a copy of the fidelity checklist
completed on the facilitators who facilitated groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of Male Healthy Relationships. The program uses internal fidelity
montoring/corrective action checklists specific to the each curriculum. The completed checklist
includes feedback specifying either the facilitator did not require any corrective action or, if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the clinical director and a review of the documentation provided verified the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found all therapists and the clinical
director received at least a ninety (90)-day or an annual evaluation within the last year. The
performance evaluationa for these staff found entries of the staff's skill in delivering the Male
Healthy Relationships bundled services. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 36 youth in the
sample, 36% (13 of 36) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 36 youth in the
sample, 100% (36 of 36) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 8 youth
High = 18 youth
Total Youth in Sample = 36 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 18 youth
Total Youth in Sample = 36 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (30 of 36) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 50% (18 of 36) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 82 95 86%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Kissimmee
Youth Academy - BDD/DD.
36 youth completed the program during the review period. Of those youth, 36 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 36 total youth sampled, only
13 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 25
and 46 weeks of service, with an average of 38 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 36 total youth
sampled, all 36 included dosage in the EBS Module. Of those youth, 36 received at least the
recommended hours of service. Youth in the sample completed between 51 and 267 hours of
service, with an average of 131 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 92%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy - BDD/DD can optimize their SPEP Quality of Service Delivery score
by ensuring facilitator turnover does not impact groups and fidelity monitoring is completed every
month for each facilitator.
Kissimmee Youth Academy - BDD/DD can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 KYA BDD-DD MHR.pdf

service_type: Male Healthy Relationships


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Kissimmee Youth Academy - BDD/DD
Youth Opportunity Investments, LLC
(Contract Provider)
2330 New Beginnings Road
Kissimme , Florida 34744
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 05/01/22-04/30/23
SPEP Review Conducted: 05/09/23
Florida Department of Juvenile Justice
Report Date: 10/24/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Skillstreaming the Adolescent intervention is being reviewed. There is a formal training
protocol which is completed by the clinical director or his designee to include training in the
Skillstreaming currciulum, group observations, and debriefings. During the review period from
05/01/2022 through 04/30/2023, there were three licensed therapists and twelve (12) non-
licensed therapists trained in the curriculum. The twelve (12) non-licensed therapists facilitated
group during the review period. The three licensed therapists provided fidelity monitoring for the
non-licensed therapists. The prior clinical director, one licensed therapist and five of the non-
licensed therapists left during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided for review was Skillstreaming the Adolescent, A Guide for Teaching
Prosocial Skills, Third Edition. There is a facilitator manual which contains a script for the
facilitator for the groups. It provides lessons and includes implementation guidelines, identifies the
order the lessons should be delivered in, and includes detailed instructions for the delivery of each
lesson.
The intervention is held on Tuesdays and Thursdays from 2:20-3:20, 3:20-4:20, and 11:08-12:08.
The program maintains their records in SPEP binders which are divided by intervention and
designation of youth as to BDD/DD, MHOS, or SAOS. All youth at Kissimmee Youth Academy
receive Skillstreaming. Skillstreaming is an open, ongoing group.
The Skillstreaming groups are held as follows: East Wing-There are two groups held on Tuesdays
and Thursdays for SAOS from 2:20-3:20 and 3:20-4:20.
There are two groups for BDD held on Tuesdays and Thursdays from 2:20-3:20 and 3:20-4:20.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Max groups are held on Tuesdays and Thursday from 2:20-3:20 and 3:20-4:20.
For West Wing, Skillstreaming is held for MHOS on Tuesdsays and Thursdays from 2:20-3:20
and from 3:20-4:20.
The BDD groups are held on Tuesdays and Thursday from 2:20-3:20.
Two groups for MAX are held on Tuesdays and Thursdays from 11:08-12:08.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed on 05/09/2023 from 11:08 to 12:08.
There were two staff and eight youth present in the group. The facilitator provided a copy of the
lesson, Expressing Affection, to the reviewer. The facilitator followed the lesson plan according to
the manual, did not read the lesson verbatim but rather used her own words to convey the
meaning of the discussed topic. She was knowledgeable of the materials and was easy for the
youth to follow. She ensured the youth understood each concept prior to moving forward in the
lesson. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation provided and an interview with the clinical director confirmed the
program has had some Skillstreaming the Adolescent facilitator turnover during the review period.
The program lost the prior clinical director, one licensed therapist and five therapists during the
review period. The other trained therapists continued facilitating the groups. The program's policy
regarding anticipated turnover is addressed by having administrative or other trained staff step in
to facilitate the group if necessary. There were no gaps in the services of the intervention to the
youth. There are currently up to nine youth in the two ongoing groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has an in-house process to perform internal fidelity monitoring for facilitators of
Skillstreaming the Adolescent. The monitoring was completed by the clinical director who is
formally trained in the curriculum, or a trained designee. The program uses internal fidelity
monitoring checklists specific to Skillstreaming to document the monitoring. The program had
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
completed checklists for all facilitators for each month they completed the groups except for the
month of 06/22 and 03/23. All completed fidelity checklists are maintained in the SPEP binder
under a tab entitled "Fidelity Monitoring for Skillstreaming". Each tab had a copy of the fidelity
checklist completed on the facilitators who facilitated groups during the specific month.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity/monitoring of Skillstreaming the Adolescent. The program uses internal fidelity
montoring/corrective action checklists specific to the curriculum. Each completed checklist
includes feedback specifying either the facilitator did not require any corrective action or, if the
facilitator did require corrective action, it would include recommendations for future improvement.
The following month's checklist would then include comments the previous month's
recommendations were applied and demonstrated.
An interview with the clinical director and a review of the documentation provided verified the
program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the current trained facilitators found these staff had received at
least either a ninety (90)-day evaluation or an annual review within the last year. The performance
evaluationsfound entries of the staff's skill in delivering Skillstreaming the Adolescent. These
included comments on any strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 35 youth in the
sample, 97% (34 of 35) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 35 youth in the
sample, 97% (34 of 35) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 8 youth
High = 17 youth
Total Youth in Sample = 35 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 17 youth
Total Youth in Sample = 35 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (29 of 35) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 49% (17 of 35) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 76 85 89%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at
Kissimmee Youth Academy - BDD/DD.
36 youth completed the program during the review period. Of those youth, 35 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 35 total youth sampled, only
34 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 34
received at least the recommended weeks of service. Youth in the sample completed between 25
and 79 weeks of service, with an average of 40 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 35 total youth
sampled, only 34 included dosage in the EBS Module. Of those youth with correct dosage, 34
received at least the recommended hours of service. Youth in the sample completed between 50
and 500 hours of service, with an average of 248 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 95%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Kissimmee Youth Academy - BDD/DD can optimize their SPEP Quality of Service Delivery score
by ensuring facilitator turnover does not impact groups and fidelity monitoring is completed every
month for each facilitator.
Kissimmee Youth Academy - BDD/DD can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 KYA BDD-DD Skillstreaming.pdf