facility_name: Center for Success and Independence - Ocala

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Centers for Success and Independence, Ocala Academy
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Seven Challenges
SPEP Review Date(s): March 11-12, 2020
Florida Department of Juvenile Justice
Report Date(s): March 12, 2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2019 through February 2020
shows the program utilizes Seven Challenges as their primary service. Seven Challenges training
is an internal training conducted by the director of clinical services, clinical director, or assistant
clinical director. The documentation showed there are twelve staff members trained in the primary
service. There are currently six staff members facilitating the primary service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Seven Challenges Manual, second edition. The workbook is broken
down into sections with activities for the youth to complete. The workbook has instructions for the
facilitators to use when facilitating the lessons. The primary service is an open group which is ran
year-round.
The first youth was admitted to the program on 1/28/2019. The director of clinical services stated
the program did not begin running the primary service until 4/8/2019. The director of clinical
services stated the reason for the gap was that the program needed to have a site visit first by the
qualified master trainer. The site visit was conducted at the end of February 2019. The program
needed to have all clinical staff in place to have the site visit.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges was observed on 3/11/2020, from 3:20p until 4:20p. The facilitator
began group by asking the youth to state the group rules. The facilitator asked a youth to conduct
the check in for the group. The youth went around the room asking the other youth how they were
doing on a scale of one to ten. The youth then asked the group to state one skill they learned at
the program and what they will do when the leave the program. The facilitator then asked a youth
to read the section on new life skills out of the Seven Challenges reader. The facilitator held an
open discussion with the group about the topic. The facilitator asked the question of the day,
“what life skill do you need to stop using substances?” The youth answered the question one at a
time. The facilitator wrapped up by asking the youth for last minute comments or questions about
the lesson.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period, the program has experienced facilitator
turnover; however, this turnover did not adversely affect delivery of Seven Challenges. The
primary service is running year-round as an open group. The program transitioned the youth to
another trained facilitator in the event of staff turnover.
The program admitted their first youth on 1/28/2019. The program did not begin running the
primary service until 4/8/2019. The director of clinical services stated the program needed a site
visit first by the master trainer before they could schedule the training. The site visit was
conducted in February 2019.
The program has a process in place to avoid gaps in service due to facilitator turnover. According
to the director of clinical services, Youth Opportunity Investments provides staff incentives to
decrease the likelihood of turnover in implementation of the Standardized Program Evaluation
Protocol services.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were twenty-one fidelity
monitorings completed for Seven Challenges during the review period. The fidelities were
completed each month except for September 2019 through January 2020.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the director of clinical services, the clinical director provides monthly fidelity
monitoring for each therapist for each primary service facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken during the
review period.
The program has a process in place to address corrective actions resulting from internal fidelity
monitoring. According to the director of clinical services, all facilitators who do not provide
adequate facilitation of Standardized Program Evaluation Protocol services will receive
progressive discipline to include retraining, coaching, and, potentially, formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed staff evaluations were completed. The evaluations have an
area to document if the facilitator has been trained in all facets of the position. The program lists
the trainings the facilitator has completed, but there is no assessment of the facilitator's
proficiency in conducting the primary service. The evaluation has a one through four scale and an
area to write in comments. The clinical director did not know the comments section should be
filled out for the facilitators of the primary service. The Technical Assistance Specialist
recommended the clinical director fill out the comment section with specific information about the
facilitator’s proficiency delivering the primary service.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement. The director of clinical
services noted the ninety-day evaluations do not have an assessment of the facilitator's
proficiency in the primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 33 youth in the
sample, 70% (23 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 33 youth in the
sample, 100% (33 of 33) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 4 youth
Moderate-High = 14 youth
High = 15 youth
Total Youth in Sample = 33 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 33 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (33 of 33) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 45% (15 of 33) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 81 95 85%
This SPEP report evaluates Seven Challenges, an intervention delivered at Centers for Success
and Independence, Ocala Academy.
58 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 33 total youth sampled, all
33 included dosage with end dates in the EBS Module. Of those youth, 23 received at least the
recommended weeks of service. Youth in the sample completed between 17 and 41 weeks of
service, with an average of 28 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 33 total youth
sampled, all 33 included dosage in the EBS Module. Of those youth, 33 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 59 and 144 hours of
service, with an average of 93 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Centers for Success and Independence, Ocala Academy, can optimize their SPEP Quality of
Service Delivery score by ensuring there are no unnecessary gaps in service, by ensuring fidelity
monitoring is completed every month for each facilitator, and by ensuring each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Centers for Success and Independence, Ocala Academy can optimize their SPEP Amount of
Service score by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring
that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

report_type: SPEP

program_status_note: Open

service_type: Seven Challenges


state: FL

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence, Ocala Academy
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Date(s): March 11-12, 2020
Florida Department of Juvenile Justice
Report Date(s): 3/12/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2019 through February 2020
shows the program utilizes Impact of Crime as their primary service. Impact of Crime training is a
four-day training conducted by a qualified trainer. The documentation showed there are seven
staff members trained in the primary service. Six trained staff members have facilitated groups
during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime Curriculum. Then the manual goes over all seven chapters
of the group and provides a script for the facilitator to follow. The manual also has an appendix
and a student guide with all the student exercises. The primary service is a closed group.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed. The groups are held after 5p on
Tuesdays and Thursdays. The reviewer had another obligation during that time.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation, the program has experienced facilitator turnover. This turnover did
not adversely affect the facilitation of Impact of Crime. One of the trained facilitators left the
program in the middle of the cohort. The program transitioned the youth to another trained
facilitator without any gaps in service. The program is utilizing a co-facilitator for the newly trained
staff.
The program is split into two different groups. One group is for the Mental Health Overlay
Services (MHOS) youth and the other is for the Substance Abuse Overlay Services (SAOS)
youth. There were four cohorts for each group of the program. The cohorts for the MHOS group
were 10/22/2019-2/11/2020, 7/9/2019-10/31/2019, 6/10/2019-9/30/2019, and 6/11/2019-
10/1/2019. The four cohorts for the SAOS group were 10/15/2019-2/4/2020, 1/20/2020-3/10/2020,
2/26/2020-6/20/2019, and 4/22/2019-8/14/2019.
The program has a process in place to address gaps in services related to facilitator turnover.An
interview with the director of clinical services revealed Youth Opportunity Investments provides
staff incentives to decrease the likelihood of turnover in implementation of the Standardized
Program Evaluation Protocol services.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation showed there were twenty-nine fidelity monitorings completed for the
review period between both the MHOS and SAOS groups. The fidelities were completed at least
monthly for each facilitator, with the exception of October 2019.
An interview with the director of clinical services revealed Impact of Crime trained staff provides
monthly fidelity monitoring to Impact of Crime facilitators and co-facilitators.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken during the
review period.
And interview with the director of clinical services confirmed the program has a process in place to
address corrective actions based on internal fidelity monitoring. All facilitators who do not provide
adequate facilitation of Standardized Program Evaluation Protocol services will receive
progressive discipline to include retraining, coaching, and, potentially, formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed the evaluations have an area to document if the facilitator has
been trained in all facets of the position. There are a few evaluations that had a numerical value
assigned for the performance of the facilitator, but there were not any comments relating to the
performance of that facilitator conducting the primary service.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 56 youth in the
sample, 57% (32 of 56) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 56 youth in the
sample, 79% (44 of 57) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 98%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 8 youth
Moderate-High = 23 youth
High = 24 youth
Total Youth in Sample = 56 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 43%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 24 youth
Total Youth in Sample = 56 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 98% (55 of 56) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 43% (24 of 56) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 25 25 100%
Totals 65 85 77%
This SPEP report evaluates Impact of Crime, an intervention delivered at Center for Success and
Independence, Ocala Academy.
58 youth completed the program during the review period. Of those youth, 56 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 56 total youth sampled, only
37 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 32
received at least the recommended weeks of service. Youth in the sample completed between 9
and 17 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Contact Hours. Of the 56 total youth
sampled, only 55 included dosage in the EBS Module. Of those youth with correct dosage, 44
received at least the recommended hours of service. Youth in the sample completed between 17
and 41 hours of service, with an average of 33 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Center for Success and Independence, Ocala Academy, can optimize their SPEP Quality of
Service Delivery score by ensuring facilitator turnover does not impact groups, by ensuring fidelity
monitoring is completed every month for each facilitator, and by ensuring each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Center for Success and Independence, Ocala Academy, can optimize their SPEP Amount of
Service score by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring
that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

program_name_raw: Center for Success and Independence - Ocala

service_type: Impact of Crime


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Centers for Success and Independence, Ocala Academy
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Moral Reconation Therapy
SPEP Review Date(s): March 11-12, 2020
Florida Department of Juvenile Justice
Report Date(s): 3/12/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2019 through February 2020
shows the program utilizes Moral Reconation Therapy (MRT) as their primary service. Moral
Reconation Therapy training conducted by a qualified trainer. The documentation showed there
are eight staff members trained in the primary service, with five staff members currently facilitating
groups. The program was only recently able to get a qualified trainer out to the program to
conduct an additional training for new staff the first week of March 2020.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Moral Reconation Therapy Facilitator’s Handbook. The Moral Reconation
Therapy Facilitator’s Handbook contains information on the group process, specific objective
criteria and guidelines for all exercises and tasks in Moral Reconation Therapy, and specific “how
to” instructions. There are separate student manuals which are given to the youth. Moral
Reconation Therapy was designed to be an open group with groups being ran year-round.
The program’s contract states the service is to be a closed group. The clinical director stated the
program was running the primary service as an open group in accordance with the service’s
implementation guidelines and not as a closed group as stated in the contract. The first youth was
admitted to the program on 1/28/2019. The program indicated they would be seeking a contract
amendment to correct the inaccuracy in the group sessions exhibit.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical assistance specialist was unable to observe the primary service. The program was not
running the groups at the time of the review. The clinical staff were new and were just trained as
of 3/5/2020. The clinical director also stated there were only seven Mental Health Overlay Service
(MHOS) youth who would qualify for the primary service. The clinical director stated there are
fifteen MHOS youth in the program; however, eight of those youth would be transitioning out of
the program within the next month or two.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period, the program has experienced facilitator
turnover which adversely affected delivery of Moral Reconation Therapy. The primary service is
running year-round as an open group. After January 2020, the only staff trained in the curriculum
was the clinical director. The clinical department now is fully staffed and all therapists are trained
in the primary service as of 3/5/2020.
The program admitted their first youth on 1/28/2019. The program did not begin running the
primary service for the Mental Health Overlay Service youth until 8/30/2019. The program did not
have any trained facilitators in the primary service until that date.
The program has a process in place to prevent gaps in service due to facilitator turnover.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were three internal fidelity
monitorings completed for the review period. The fidelity monitorings were completed on
1/3/2020, 11/22/2019, and 9/20/2019. The program is missing internal fidelity checks for the
months of August, October, and December 2019. The program was not running the primary
service prior to 8/30/2019 or after January 2020.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Accoring to the director of clinical services, the clinical director provides monthly fidelity monitoring
for each therapist for each primary service facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken during the
review period.
The program has a process in place to address corrective actions resulting from internal fidelity
monitoring. According to the director of clinical services, all facilitators who do not provide
adequate facilitation of Standardized Program Evaluation Protocol services will receive
progressive discipline to include retraining, coaching, and potentially formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed staff evaluations were completed. The evaluations have an
area to document if the facilitator has been trained in all facets of the position. The program lists
the trainings the facilitator has completed, but there is no assessment of the facilitator's
proficiency in conducting the primary service, Moral Reconation Therapy. The evaluation has a
one through four scale and an area to write in comments. The clinical director did not know the
comments section should be filled out for the facilitators of the primary service. The Technical
Assistance Specialist recommended the clinical director fill out the comment section with specific
information about the facilitator’s proficiency delivering the primary service.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement. The director of clinical
services noted the ninety-day evaluations do not have an assessment of the facilitator's
proficiency in the primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 22 youth in the
sample, 36% (8 of 22) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 22 youth in the
sample, 0% (0 of 22) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 4 youth
Moderate-High = 9 youth
High = 8 youth
Total Youth in Sample = 22 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 36%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 22 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (21 of 22) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 36% (8 of 22) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 72 100 72%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at Centers for
Success and Independence, Ocala Academy.
58 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 22 total youth sampled,all
22 included dosage with end dates in the EBS Module. Of those youth, 8 received at least the
recommended weeks of service. Youth in the sample completed between 5 and 22 weeks of
service, with an average of 13 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 22 total youth
sampled, all 22 included dosage in the EBS Module. Of those youth, 0 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 4 and 40 hours of
service, with an average of 21 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Centers for Success and Independence, Ocala Academy, can optimize their SPEP Quality of
Service Delivery score by ensuring there are no unnecessary gaps in service, by ensuring fidelity
monitoring is completed every month for each facilitator, and by ensuring each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Centers for Success and Independence, Ocala Academy, can optimize their SPEP Amount of
Service score by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring
that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Moral Reconation Therapy


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Centers for Success and Independence, Ocala Academy
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Skillstreaming the Adolescent
SPEP Review Date(s): 3/11/2020-3/12/2020
Florida Department of Juvenile Justice
Report Date(s): 3/12/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2019 through February 2020
shows the program utilizes Skillstreaming the Adolescent as their primary service. Skillstreaming
the Adolescent training is an internal training conducted by the director of clinical services, clinical
director, or assistant clinical director. The documentation showed there are twelve staff members
trained in the primary service. There are currently seven staff members facilitating the primary
service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes Skillstreaming the Adolescent 3rd edition. The manual introduces
Skillstreaming, contains scripts for the facilitators to follow as well as specific instructions for each
skill taught, example role plays, and student exercises. There are no separate student manuals.
The facilitator makes copies of the skill cards for each skill and hands out those copies to the
youth in the group. There are fifty skills that are taught to the youth. Skillstreaming is an open
group with groups being ran year-round. The first youth was admitted to the program on
1/28/2019. The first attendance sheet for the Substance Abuse Overlay Service group was in
February 2019. The first attendance sheet for the Mental Health Overlay Service group was in
April 2019.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Technical assistance specialist observed the primary service on 3/11/2020. The group began at
4:23p and ended at 5:18p. The facilitator asked the youth about the group rules. The facilitator
then asked two youth to conduct the group check in. The facilitator stated that it was role play day.
The facilitator stated that the skill of the day is knowing your feelings. The facilitator passed out
the skill cards to each of the youth. The facilitator then asked the youth to read each step on the
skill card. The facilitator asked the youth to identify each step as a thinking step or an action step.
The youth were asked to pair up with another youth. Each group came up with their own role play.
The youth finished their role plays quickly. The facilitator asked the youth to conduct another
round of role play. The youth were happy to conduct another role play. The facilitator realized that
she did not demonstrate the skill to the youth prior to the role plays. The facilitator asked for two
volunteers to help complete the role play demonstration. The facilitator asked the youth how they
tune into their body to know how they are feeling and why is this step so important? The facilitator
told the youth two takeaways that the youth should have at the end of the session. Group ended.
Technical assistance specialist spoke with the facilitator about the observation. Technical
assistance specialist informed the facilitator that she did well. Technical assistance specialist
advised that the facilitator should assign the skill steps to the youth prior to demonstrating the role
play. Technical assistance specialist also recommended that the youth state the step that they
were given to look for as well as define the step in their own words.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to the documentation for the annual review period, the program has experienced
facilitator turnover. This turnover did not adversely affect delivery of the primary service. The
primary service is running year-round as an open group. The program transitioned the youth to
another trained facilitator in the event of staff turnover.
The program admitted their first youth on 1/28/2019. The program did not begin running the
primary service for the Substance Abuse Overlay Service youth until February 2019. The program
did not begin running the primary service for the Mental Health Overlay Service youth until April
2019. There were no sign in sheets available from 9/24/2019 through 10/1/2019.
Per the director of clinical services, Youth Opportunity Investments provides staff incentives to
decrease the likelihood of turnover in implementation of the Standardized Program Evaluation
Protocol services.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A review of the documentation for the annual review period showed that there were twenty-one
fidelity monitoring’s completed for the review period. The fidelities were completed on 2/20/2020,
2/13/2020, 2/4/2020, 1/7/2020, 12/19/2020, 12/17/2019, 12/10/2019, 3/25/2019, 4/2/2019,
4/4/2019, 4/23/2019, 5/14/2019, 5/23/2019, 5/23/2019, 6/11/2019, 6/11/2019, 6/13/2019,
7/2/2019, 7/16/2019, 7/4/2019, and 7/25/2019. The program is missing internal fidelity checks for
the months of February 2019, September 2019, October 2019, and November 2019.
Per the director of clinical services, the clinical director provides monthly fidelity monitoring for
each therapist for each primary service that is facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation showed that there were no corrective actions noted.
Per the director of clinical services, all facilitator’s who do not provide adequate facilitation of
Standardized Program Evaluation Protocol services will receive progressive discipline to include
retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the documentation showed that there were staff evaluations completed. The
evaluations have an area to document if the facilitator has been trained in all facets of the
position. The program lists the trainings that the facilitator has completed, but there is no
assessment of the facilitator's proficiency in conducting the primary service. The evaluation has a
one through four scale and an area to write in comments. The clinical director did not know that
the comments section should be filled out for the facilitators of the primary service. Technical
assistance specialist recommended that the clinical director fill out the comment section with
specific information about the facilitator’s proficiency delivering the primary service.
Per the director of clinical services, all facilitators are evaluated yearly on group services provided.
Evaluations include implementation and youth engagement. The director of clinical services noted
that the ninety-day evaluations do not have an assessment of the facilitator's proficiency in the
primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 57 youth in the
sample, 74% (42 of 57) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 57 youth in the
sample, 84% (48 of 57) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 98%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 8 youth
Moderate-High = 24 youth
High = 24 youth
Total Youth in Sample = 57 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 24 youth
Total Youth in Sample = 57 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 98% (56 of 57) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 42% (24 of 57) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 69 85 81%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Centers
for Success and Independence, Ocala Academy.
58 youth completed the program during the review period. Of those youth, 57 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 57 total youth sampled, only
28 included dosage with end dates in the EBS Module, as a result of an administrative oversight.
A manual audit of the remaining youth was conducted to include dosage calculations. Of those
youth, 42 received at least the recommended weeks of service. Youth in the sample completed
between 10 and 33 weeks of service, with an average of 21 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 57 total youth
sampled, only 28 included dosage in the EBS Module, as a result of an administrative oversight. A
manual audit of the remaining youth was conducted to include dosage calculations. Of those
youth, 48 received at least the recommended hours of service. Youth in the sample completed
between 17 and 59 hours of service, with an average of 39 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Centers for Success and Independence, Ocala Academy, can optimize their SPEP Quality of
Service Delivery score by ensuring there are no unnecessary gaps in service, by ensuring fidelity
monitoring is completed every month for each facilitator, and by ensuring each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Centers for Success and Independence, Ocala Academy, can optimize their SPEP Amount of
Service score by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring
that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Skillstreaming the Adolescent


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Seven Challenges
SPEP Review Period: 2/1/2020-1/31/2021
SPEP Review Conducted: 3/16/2021
Florida Department of Juvenile Justice
Report Date: 4/2/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 2020 through January 2021
shows the program utilizes Seven Challenges as their primary service. Seven Challenges training
is an internal training conducted by the director of clinical services, clinical director, or assistant
clinical director. The director of clinical services, clinical director, and assistant clinical director
completed the Seven Challenges Leader Training in order to conduct the Seven Challenges staff
training. The documentation showed there are seven staff members trained in the primary service.
The trained staff include the clinical director, assistant clinical director, and five therapists. All
seven staff members have facilitated the primary service during the annual review period. The
director of clinical services, clinical director, or assistant clinical director conduct the monthly
internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period shows the program utilizes the Seven
Challenges Manual, second edition. The program is licensed to facilitate the Seven Challenges
curriculum. The workbook is broken down into sections which have activities for the youth to
complete. The workbook also has instructions for the facilitators to use when facilitating the
lessons. The primary service is an open group which is run year-round.
The service is provided in two different groups. One group is for the Mental Health Overlay
Services (MHOS) youth and the other is for the Substance Abuse Overlay Services (SAOS)
youth. The program maintains the attendance sheets and monthly internal fidelity checks in the
SPEP binder titled "Seven Challenges".
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges was observed on 9/8/2020 via FaceTime due to COVID-19.
Group began at 11:35a. All youth and staff had their facemasks on. The youth were in one of the
classroom portables for group. The facilitator began group by having one youth ask the other
youth their daily check-in questions. The youth are able to decide at the beginning of group what
topic they will be addressing during their session. Youth recited the group rules to the facilitator
one at a time. Youth appeared to be respectful to the facilitator and raised their hands if they
wanted to speak. The group discussed how drugs affect brain development. They also discussed
patience and labels given the youth like addict. Each youth took turns reading out of the Seven
Challenges reader. The facilitator then asked the youth what they got out of each passage read.
The facilitator ended the group by asking each youth “What reasons do you have to talk about
substance use?” Group ended at 12:20p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period, the program has experienced facilitator
turnover. The clinical director, assistant clinical director, and a therapist left the program during
the annual review period. The program continued Seven Challenges by transitioning the youth to
another trained facilitator. There were no gaps in service. The primary service is running year-
round as an open group.
According to the director of clinical services, there were three out of seven mental health
professionals who facilitated groups during the youth quarantine which began in July 2020 and
lasted approximately three to four weeks. There were four of the seven mental health
professionals who were unable to facilitate groups during that time due to COVID-19. Those four
staff members came back and took over the groups once they were allowed.
Per the director of clinical services, Youth Opportunity Investments provides staff incentives to
decrease the likelihood of turnover in implementation of the Standardized Program Evaluation
Protocol services.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A review of documentation showed there were nineteen fidelity monitorings completed for the
review period. The program is missing internal fidelity checks for the months of August 2020,
October 2020, December 2020, and January 2021 for three of the therapists. The program is
missing one internal fidelity check for one therapist in September 2020. The program is missing
internal fidelity checks in November 2020 and March 2020.
The clinical director wrote a note on a fidelity form for 7/20/2020. The note stated: In light of
quarantine status at CSI-Ocala and COVID-19 outbreak, we are unable to run formal groups.
Youth are receiving group treatment interventions through supportive counseling measures.
According to the director of clinical services, the clinical director provides monthly fidelity
monitoring for each therapist for each primary service that is facilitated by the therapists.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted nor taken.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed staff evaluations were completed. The evaluations have an
area to document if the facilitator has been trained in all facets of the position. The evaluation has
a one through four scale and an area to write in comments. Of the seven staff who facilitated
Seven Challenges groups, only four had comments regarding the staffs' skill in delivering Seven
Challenges.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement. The director of clinical
services noted the ninety-day evaluations do not have an assessment of the facilitator's
proficiency in the primary service, but the yearly evaluations do.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 43 youth in the
sample, 86% (37 of 43) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 43 youth in the
sample, 100% (43 of 43) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 17 youth
High = 15 youth
Total Youth in Sample = 43 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 35%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 43 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (38 of 43) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 35% (15 of 43) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 81 95 85%
This SPEP report evaluates Seven Challenges, an intervention delivered at Center for Success
and Independence - Ocala.
71 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 43 total youth sampled, only
38 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 37
received at least the recommended weeks of service. Youth in the sample completed between 19
and 50 weeks of service, with an average of 32 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 43 total youth
sampled, all 43 included dosage in the EBS Module. Of those youth, 43 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 44 and 219 hours of
service, with an average of 102 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups, fidelity monitoring is
completed every month for each facilitator, and each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9