summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Period: 2/1/2020-1/31/2021
SPEP Review Conducted: 3/17/2021
Florida Department of Juvenile Justice
Report Date: 4/2/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period shows the program utilizes Impact of
Crime as their primary service. Impact of Crime training is four-days, conducted by a qualified
trainer. The documentation showed there are six staff members trained in the primary service. All
six trained staff members have facilitated groups during the annual review period. The trained
staff members are three case managers, one transition services manager, one transitional case
manager, and the director of case management. The director of case management and one case
manager conducted the monthly internal fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime Curriculum. Then the manual goes over all seven chapters
of the group and provides a script for the facilitator to follow. The manual also has an appendix
and a student guide with all the student exercises. The primary service is a closed group.
The program is split into two different groups. One group is for the Mental Health Overlay
Services (MHOS) youth and the other is for the Substance Abuse Overlay Services (SAOS)
youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
A session of Impact of Crime was observed on 9/15/2020 via FaceTime due to COVID-19. Group
began at 4:20p. The facilitator went over the group rules with the youth. There were two youth
who did not have their facemasks on at the beginning of the group. Both youths put their masks
on after being prompted. The facilitator went into an icebreaker with the youth asking them
several riddles. The youth seemed to enjoy the icebreaker. The facilitator then conducted a pop
quiz, asking questions about what the youth learned during chapter one. The facilitator went over
the answers with all of the youth once they were finished with the quiz. The facilitator began
chapter two by going over the vocabulary for the chapter. Some of the youth had their heads
down on the table. The other youth were actively participating. The facilitator told the youth she
would bring them paper and pencils for the next class so they could take notes. The facilitator
ended group at 5:18p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed showed the program has experienced facilitator turnover. This
turnover adversely affected the facilitation of Impact of Crime. One of the trained facilitators left
the program in the middle of the cohort. The program transitioned the youth to another trained
facilitator without any gaps in service. The program was utilizing a co-facilitator for the newly
trained staff.
The documentation reviewed for the annual review period showed the program completed three
cohorts for the SAOS youth and two cohorts for the MHOS youth. The dates of the cohorts were
3/24/2020-7/14/2020, 1/20/2020-5/12/2020, 3/24/2020-7/16/2020, 9/9/2020-12/31/2020, and
9/9/2020-12/28/2020.
The cohort which began on 3/24/2020 and ended on 7/14/2020 started with twelve youth and
ended with ten. One youth was arrested and the other was released. The facilitator documented
one-on-one sessions with the youth who was released to ensure the youth completed the cohort.
The cohort which began on 1/20/2020 began with twelve youth and ended with eleven. One youth
was released. This cohort began with one case manager and was transitioned to another during
the cohort due to facilitator turnover.
The cohort which began on 3/24/2020 and ended on 7/16/2020 began with twelve youth and
ended with eleven. One youth had an end date of 6/9/2020 with no documentation why the youth
did not complete the primary service.
The cohort which began on 9/9/2020 and ended on 12/28/2020 began with ten youth and ended
with nine. The youth was sent to jail.
The cohort which began on 9/9/2020 and ended on 12/31/2020 began with eleven youth and
ended with seven. Three youth were released and one youth went to jail.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation showed there were nineteen (19) fidelity monitorings completed for the
review period. The program is missing internal fidelity checks for the months of August, October,
and December 2020, and January 2021, for three of the therapists. The program is missing one
internal fidelity check for one therapist in September 2020. The program is missing internal fidelity
checks in November 2020 and March 2020.
The clinical director wrote a note on a fidelity form for 7/20/2020. The note stated: In light of
quarantine status at CSI-Ocala and COVID-19 outbreak, we are unable to run formal groups.
Youth are receiving group treatment interventions through supportive counseling measures.
Per the director of clinical services, the clinical director provides monthly fidelity monitoring for
each therapist for each primary service facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation showed there were no corrective actions noted.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the evaluations have an area to document if the facilitator has
been trained in all facets of the position. Three evaluations had a numerical value assigned for the
performance of the facilitator, but there were not any comments relating to the performance of that
facilitator conducting the primary service. One of the evaluations had a statement about the staff
member’s understanding of material and facilitating groups, but did not contain any other
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
information. Two evaluations contained specific remarks relating to the facilitator’s delivery of
Impact of Crime.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 70 youth in the
sample, 71% (50 of 70) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 70 youth in the
sample, 86% (60 of 70) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 29 youth
High = 28 youth
Total Youth in Sample = 70 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 28 youth
Total Youth in Sample = 70 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (63 of 70) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 40% (28 of 70) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Center for Success and
Independence - Ocala.
71 youth completed the program during the review period. Of those youth, 70 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 70 total youth sampled, only
64 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 50
received at least the recommended weeks of service. Youth in the sample completed between 4
and 17 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 70 total youth
sampled, only 68 included dosage in the EBS Module. Of those youth with correct dosage, 60
received at least the recommended hours of service. Youth in the sample completed between 8
and 41 hours of service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups, fidelity monitoring is
completed every month for each facilitator, and each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Moral Reconation Therapy
SPEP Review Period: 2/1/2020-1/31/2021
SPEP Review Conducted: 3/18/2021
Florida Department of Juvenile Justice
Report Date: 4/7/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2020 through January 2021
shows the program utilizes Moral Reconation Therapy (MRT) as their primary service. Moral
Reconation Therapy training is conducted by a qualified trainer. The documentation showed there
are seven staff members trained in MRT and all seven facilitated the primary service during the
annual review period. The clinical director, assistant clinical director, or state director of clinical
services conducts the monthly internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Moral Reconation Therapy Facilitator’s Handbook. The Moral Reconation
Therapy Facilitator’s Handbook contains information on the group process, specific objective
criteria and guidelines for all exercises and tasks in Moral Reconation Therapy, and specific “how
to” instructions. There are separate student manuals given to the youth. Moral Reconation
Therapy is designed to be an open group with groups being run year-round. The program’s
contract states the service is to be a closed group. The former clinical director stated the program
was running the primary service as an open group per the service’s implementation guidelines
and not as a closed group as stated in the contract.
The service is provided in two different groups. One group is for the Mental Health Overlay
Services (MHOS) youth and the other is for the Substance Abuse Overlay Services (SAOS)
youth. The program maintains the attendance sheets and monthly internal fidelity checks in the
SPEP binder titled "MRT".
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Moral Reconation Therapy (MRT) was observed on 9/9/2020 via FaceTime due to
COVID-19. Group began at 3:25p. There was one youth who did not have his mask on. The
technical assistance specialist asked the clinical director about the mask. The clinical director
informed the technical assistance specialist a staff member was going to the dorm to retrieve the
youth’s mask. The youth left it in the dorm in the morning. The facilitator passed out the MRT
books and pencils to the youth. The facilitator asked the group to go over the group expectations.
The youth presented the circle of relationship step 5. The facilitator would walk from youth to
youth to look at their handbook and ask them questions about the activity. One of the activities
was to interview a peer. The group ended at 4:15p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period, the program has experienced facilitator
turnover. The clinical director, assistant clinical director, and a therapist left the program during
the annual review period. The program continued Moral Reconation Therapy (MRT) groups by
transitioning the youth to another trained facilitator. The primary service is running year-round as
an open group.
According to the state director of clinical services, the program did not have any Moral Reconation
Therapy (MRT) groups during the month of April 2020. The program restarted the MRT groups in
May 2020.
Also according to the state director of clinical services, Youth Opportunity Investments provides
staff incentives to decrease the likelihood of turnover in implementation of the Standardized
Program Evaluation Protocol services.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were nine fidelity
monitorings completed for the review period. There was a fidelity check completed for the month
of May for one therapist, but the form did not have the specific date the fidelity was performed.
The program is missing internal fidelity checks for the months of November 2020, December
2020, and January 2021 for all four therapists. The program is missing one internal fidelity check
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
for one therapist in May 2020 and September 2020. The program is missing two internal fidelity
checks for two therapists for June 2020, July 2020, and October 2020. The program is missing
internal fidelity checks for three therapists in August 2020.
The clinical director wrote a note on a fidelity form for 7/20/2020. The note stated: "Fidelity
monitoring is limited due to precautionary measures being taken to guard against the spread of
COVID-19. Observed four fifteen-minute supportive counseling sessions."
According to the state director of clinical services, the clinical director provides monthly fidelity
monitoring for each therapist for each primary service facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken during the
review period.
According to the state director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period showed staff were evaluations completed.
The evaluations have an area to document if the facilitator has been trained in all facets of the
position. The evaluation has a one through four scale and an area to write in comments. Of the
seven facilitators, four had evaluations with remarks on their skill in delivering Moral Reconation
Therapy, while three had only a numerical value assigned.
According to the state director of clinical services, all facilitators are evaluated yearly on group
services provided. Evaluations include implementation and youth engagement. The state director
of clinical services noted the ninety-day evaluations do not have an assessment of the facilitator's
proficiency in the primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 28 youth in the
sample, 96% (27 of 28) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 28 youth in the
sample, 57% (16 of 28) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 13 youth
High = 13 youth
Total Youth in Sample = 28 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 46%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 13 youth
Total Youth in Sample = 28 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (26 of 28) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 46% (13 of 28) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 80 100 80%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at Center for
Success and Independence - Ocala.
71 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 28 total youth sampled, all
28 included dosage with end dates in the EBS Module. Of those youth, 27 received at least the
recommended weeks of service. Youth in the sample completed between 14 and 45 weeks of
service, with an average of 27 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 28 total youth
sampled, all 28 included dosage in the EBS Module. Of those youth, 16 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 19 and 83 hours of
service, with an average of 49 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups, fidelity monitoring is
completed every month for each facilitator, and each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_status_note: Open

service_type: Moral Reconation Therapy


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 2/1/2020-1/31/2021
SPEP Review Conducted: 3/16/2021
Florida Department of Juvenile Justice
Report Date: 4/2/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2020 through January 2021
shows the program utilizes Skillstreaming the Adolescent as their primary service. Skillstreaming
the Adolescent training is an internal training conducted by the director of clinical services, clinical
director, or assistant clinical director. There are four therapists trained in the primary service. All
four staff members have facilitated the primary service during the annual review period. The
program did not provide proof of training for one therapist and the assistant clinical director. The
director of clinical services, clinical director, or assistant clinical director conduct the monthly
internal fidelity monitoring.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the 3rd edition of Skillstreaming the Adolescent. The manual introduces
Skillstreaming, contains scripts for the facilitators to follow as well as specific instructions for each
skill taught, example role plays, and student exercises. There are no separate student manuals.
The facilitator makes copies of the skill cards for each skill and hands out those copies to the
youth in the group. There are fifty skills taught to the youth. Skillstreaming is an open group, with
groups being run year-round. Skillstreaming is held on Tuesdays and Thursdays at 3:20p for one
hour.
The service is provided in two different groups. One group is for the Mental Health Overlay
Services (MHOS) youth and the other is for the Substance Abuse Overlay Services (SAOS)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
youth. The program maintains the attendance sheets and monthly internal fidelity checks in the
SPEP binder titled "Skillstreaming".
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed via FaceTime on 9/10/2020, due to
COVID-19. The group began at 11:33a. The facilitator went over the group rules. One youth did
not have his mask on. The Technical Assistance Specialist asked about the youth’s mask. The
youth pulled the mask out of his pocket and put it on. One youth was chosen to conduct the check
in. The lesson is Understanding the Feelings of Others. The facilitator explained what the lesson
was about. The facilitator used an example of the youth listening to each other during group check
in. The facilitator demonstrated the skill with a role play and said she would go over the steps of
the skill afterwards. The facilitator asked a youth to read out each step and explain whether it was
a thinking step or an action step. The facilitator divided the youth into groups of two. The youth
were then asked to role play the step. Each youth were asked to be the main actor and co-actor.
The facilitator asked the group if there was any feedback about the skill learned.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation provided, the program has experienced facilitator turnover; however,
this turnover did not adversely affect delivery of Skillstreaming the Adolescent. The clinical
director and assistant clinical director left the program during the annual review period. There
were no gaps in service. The primary service is running year-round as an open group.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation showed there were twenty-five fidelity monitorings completed for
Skillstreaming the Adolescent. The program is missing one internal fidelity check for one therapist
in June 2020. The program is missing internal fidelity checks for two therapists in July 2020. The
program is missing internal fidelity checks for the month of August 2020 for three therapists. The
program is missing internal fidelity checks for the months of September 2020, December 2020,
and January 2021 for two therapists. The program is missing internal fidelity checks for one
therapist in October 2020. The program does not have any fidelity checks completed for
November 2020.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the director of clinical services, the clinical director provides monthly fidelity
monitoring for each therapist for each primary service facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken for
Skillstreaming the Adolescent.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation provided showed staff evaluations were completed. The evaluations
have an area to document if the facilitator has been trained in all facets of the position. The
evaluation has a one through four scale and an area to write in comments. Of the four facilitators
of Skillstreaming the Adolescent, only two had evaluations with feedback regarding their skill in
delivering the service.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement. The director of clinical
services noted the ninety-day evaluations do not have an assessment of the facilitator's
proficiency in the primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 71 youth in the
sample, 94% (67 of 71) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 10%
Research indicates a target of 24 contact hours for this type of service. Of the 71 youth in the
sample, 100% (71 of 71) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 30 youth
High = 28 youth
Total Youth in Sample = 71 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 28 youth
Total Youth in Sample = 71 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (64 of 71) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 39% (28 of 71) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 71 85 84%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Center for
Success and Independence - Ocala.
71 youth completed the program during the review period. Of those youth, 71 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 71 total youth sampled, only
68 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 67
received at least the recommended weeks of service. Youth in the sample completed between 19
and 50 weeks of service, with an average of 31 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 71 total youth
sampled, all 71 included dosage in the EBS Module. Of those youth, 71 received at least the
recommended hours of service. Youth in the sample completed between 35 and 84 hours of
service, with an average of 53 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring all facilitators receive formal training prior to delivering services to
youth, fidelity monitoring is completed every month for each facilitator, each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Skillstreaming the Adolescent


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Seven Challenges
SPEP Review Period: February 1, 2021-January 31, 2022
SPEP Review Conducted: March 1, 2022
Florida Department of Juvenile Justice
Report Date: 5/23/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 1, 2021 through January 31,
2022 shows the program utilizes Seven Challenges as their primary service. Seven Challenges
training is an internal training conducted by the director of clinical services, clinical director, or
assistant clinical director. The director of clinical services, clinical director, and assistant clinical
director completed the Seven Challenges Leader Training in order to conduct the Seven
Challenges staff training. The documentation showed there are seven staff members trained in
the primary service- five therapists, the current clinical director, and the assistant clinical director.
Six of the seven facilitated the primary service during the annual review period. The former clinical
director also facilitated the primary service; however, the program was unable to provide his proof
of training. The director of clinical services, clinical director, or assistant clinical director conduct
monthly internal fidelity monitoring.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period shows the program utilizes the Seven
Challenges Manual, second edition. The program is licensed to facilitate the Seven Challenges
curriculum. The workbook is broken down into sections with activities for the youth to complete.
The workbook also has instructions for the facilitators to use when facilitating the lessons. The
primary service is an open group which is run year-round in two different cohorts. One group is for
the Mental Health Overlay Services (MHOS) youth and the other is for the Substance Abuse
Overlay Services (SAOS) youth. All youth assigned to the SAOS side of the program are required
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
to complete the primary service. The program maintains the attendance sheets and monthly
internal fidelity checks in the SPEP binder titled Seven Challenges. The program provides the
primary service on Mondays and Fridays at 4:30p.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges was observed on 2/18/2022 via Zoom. The group began at 3:23p.
The group was held in one of the program’s classrooms. There were twelve youth, one facilitator,
two floor staff, and the assistant clinical director present. The facilitator was knowledgeable of the
material and followed the lesson plan. The facilitator began with the group check in and went over
the group rules. The youth were starting to get restless toward the end of group. There were two
youth at the end of the group who got up and wouldn’t listen to the facilitator. The facilitator tried
to redirect the youth. The group ended at 4:26p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to the documentation for annual review period of February 1, 2021 through January 31,
2022, the program has experienced facilitator turnover. There were two therapists and the former
clinical director who left in November 2021. Both therapists who left were facilitating Seven
Challenges groups. The program transitioned the youth to another trained facilitator without any
gaps in service. There were no more than fifteen youth in a group per session, as indicated by the
attendance sheets. The primary service is running year-round as an open group.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services. The director of case management stated the program will pull a co-
facilitator from one cohort in order to cover another cohort if a facilitator leaves the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were eight fidelity checks
completed for facilitators of Seven Challenges. The dates of the fidelity monitoring were
9/10/2021, 9/13/2021, 9/13/2021, 7/12/2021, 7/12/2021, 6/21/2021, 6/25/2021, and 6/25/2021.
The director of clinical services stated the program could not locate the monthly internal fidelity
checks for the months of January 2021, February 2021, and March 2021. The director of clinical
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
services also stated the program did complete monthly internal fidelity checks in April 2021, but
did not provide proof. The program did not complete any internal fidelity checks for the months of
May 2021, August 2021, October 2021, November 2021, December 2021, and January 2022. The
program was also missing monthly fidelity checks for the one therapist for June 2021, two
therapists for July 2021, and two therapists and the former clinical director in September 2021.
The former clinical director and the current clinical director conducted the fidelity checks during
the annual review period.
According to the director of clinical services, the director of case management or another trained
facilitator conducts the monthly internal fidelity checks for the primary service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted during the annual
review period for facilitators of Seven Challenges.
According to the director of clinical services, all facilitator’s who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period showed the evaluations have an area to
document if the facilitator has been trained in all facets of the position. The program provided
annual evaluations for three therapists and a ninety-day evaluation for one therapist. The annual
evaluations had a numerical value assigned for the performance of the facilitator as well as
detailed comments relating to the performance of that facilitator conducting Seven Challenges.
The ninety-day evaluation did not mention if the therapist facilitated any primary services, nor did
it have any detailed information regarding any facilitation.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 10%
Research indicates the target duration of 24 weeks for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 97%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 5 youth
Moderate-High = 9 youth
High = 15 youth
Total Youth in Sample = 30 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 97% (29 of 30) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 50% (15 of 30) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 85 95 89%
This SPEP report evaluates Seven Challenges, an intervention delivered at Center for Success
and Independence - Ocala.
75 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 30 total youth sampled, all
30 included dosage with end dates in the EBS Module. Of those youth, 30 received at least the
recommended weeks of service. Youth in the sample completed between 25 and 63 weeks of
service, with an average of 32 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 30 total youth
sampled, all 30 included dosage in the EBS Module. Of those youth, 30 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 70 and 202 hours of
service, with an average of 106 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring all facilitators receive formal training prior to delivering services to
youth, facilitator turnover does not impact groups, and fidelity monitoring is completed every
month for each facilitator.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 CSI-Ocala 7C.pdf

service_type: Seven Challenges


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: February 1, 2021-January 31, 2022
SPEP Review Conducted: March 1, 2022
Florida Department of Juvenile Justice
Report Date: 5/24/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 1, 2021 through January 31,
2022 shows the program utilizes Skillstreaming the Adolescent as their primary service.
Skillstreaming the Adolescent training is an internal training conducted by the director of clinical
services, clinical director, or assistant clinical director. The documentation showed there are nine
staff members trained in the primary service- six therapists, the assistant clinical director, the
former clinical director, and the current clinical director. Seven staff facilitated the service during
the annual review period, six therapists and the assistant clinical director. The former clinical
director and current clinical director conducted the monthly internal fidelity monitoring during the
annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Skillstreaming the Adolescent 3rd edition. The manual introduces
Skillstreaming, contains scripts for the facilitators to follow as well as specific instructions for each
skill taught, example role plays, and student exercises. There are no separate student manuals.
The facilitator makes copies of the skill cards for each skill and hands out those copies to the
youth in the group. There are fifty skills taught to the youth. Skillstreaming is an open group with
groups being run year-round. The program is split into two different groups. One group is for the
Mental Health Overlay Services (MHOS) youth and the other is for the Substance Abuse Overlay
Services (SAOS) youth. All youth are required to complete the primary service. The program
maintains the attendance sheets and monthly internal fidelity checks in the SPEP binder titled
Skillstreaming. Skillstreaming is held on Tuesdays and Thursdays at 3:20p for one hour.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed on 2/17/2022 via Zoom. The group
began at 3:32p and was held in one of the program’s classrooms. There were ten youth, one
facilitator, and two staff present. The facilitator was knowledgeable of the material and followed
the lesson plan. The facilitator began with the group check in. During check in, one youth said he
wanted an abuse call. Another youth said he was refused a grievance for having staff take his
shoes. The technical assistance specialist informed the assistant clinical director about the two
youth’s issues. There was one youth in the corner of the room who had his chair leaned back
against the door of the classroom. There was a staff member in the group who did not say
anything to that youth. The staff member looked at the youth several times during the group. The
facilitator passed out the skill cards for the Understanding the Feelings of Others skill. The youth
were having side conversations during group. The facilitator tried to redirect the youth. The group
ended at 4:30p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period of February 1, 2021 through January 31,
2022, the program has experienced facilitator turnover. There were two therapists and the former
clinical director who left in November 2021. Both therapists who left were facilitating
Skillstreaming the Adolescent. The program transitioned the youth to another trained facilitator
without any gaps in service. There were no more than ten youth in a group per session as
indicated by the attendance sheets. The primary service is running year-round as an open group.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services. The director of case management stated the program will pull a co-
facilitator from one cohort in order to cover another cohort if a facilitator leaves the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were thirteen fidelity checks
completed for facilitators of Skillstreaming the Adolescent. The director of clinical services stated
the program could not locate the monthly internal fidelity checks for the months of January 2021,
February 2021, and March 2021. The director of clinical services also stated the program did
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
complete monthly internal fidelity checks in April 2021, but did not provide proof. The program did
not complete any internal fidelity checks for the months of May 2021, July 2021, August 2021,
October 2021, and November 2021. The program was missing monthly fidelity checks for the
month of September 2021 for two therapists; December 2021 for one therapist and the assistant
clinical director; and January 2022 for two therapists and the assistant clinical director. The former
clinical director and the current clinical director conducted the fidelity checks during the annual
review period.
According to the director of clinical services, the director of case management or another trained
facilitator conducts the monthly internal fidelity checks for the primary service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted during the annual
review period for facilitators of Skillstreaming the Adolescent.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period showed the evaluations have an area to
document if the facilitator has been trained in all facets of the position. The program provided
annual evaluations for three therapists and a ninety-day evaluation for one therapist. The annual
evaluations had a numerical value assigned for the performance of the facilitator as well as
detailed comments relating to the performance of that facilitator conducting Skillstreaming the
Adolescent. The ninety-day evaluation did not mention if the therapist facilitated any primary
services, nor did it have any detailed information regarding any facilitation.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 71 youth in the
sample, 100% (71 of 71) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 71 youth in the
sample, 100% (71 of 71) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 10 youth
Moderate-High = 30 youth
High = 24 youth
Total Youth in Sample = 71 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 34%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 24 youth
Total Youth in Sample = 71 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (64 of 71) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 34% (24 of 71) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 80 85 94%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Center for
Success and Independence - Ocala.
75 youth completed the program during the review period. Of those youth, 71 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 71 total youth sampled, all
71 included dosage with end dates in the EBS Module. Of those youth, 71 received at least the
recommended weeks of service. Youth in the sample completed between 18 and 63 weeks of
service, with an average of 30 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 71 total youth
sampled, all 71 included dosage in the EBS Module. Of those youth, 71 received at least the
recommended hours of service. Youth in the sample completed between 50 and 100 hours of
service, with an average of 55 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups and fidelity monitoring is
completed every month for each facilitator
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 CSI-Ocala Skillstreaming.pdf