summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Moral Reconation Therapy
SPEP Review Period: February 1, 2021-January 31, 2022
SPEP Review Conducted: March 1, 2022
Florida Department of Juvenile Justice
Report Date: 5/24/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 1, 2021 through January 31,
2022 shows the program utilizes Moral Reconation Therapy (MRT) as their primary service. Moral
Reconation Therapy training is conducted by a qualified trainer either in person or online. The
documentation shows there are seven staff members trained in MRT- four therapists, the former
clinical director, the current clinical director, and the assistant clinical director. Six of the seven
trained staff have facilitated MRT during the annual review period (four therapists, the former
clinical director, and the assistant clinical director). The current clinical director and the former
clinical director conducted the monthly fidelity monitoring during the annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Moral Reconation Therapy Facilitator’s Handbook. The Moral Reconation
Therapy Facilitator’s Handbook contains information on the group process, specific objective
criteria and guidelines for all exercises and tasks in Moral Reconation Therapy (MRT), and
specific “how to” instructions. There are separate student manuals for the youth. MRT was
designed to be an open group with groups being run year-round. The program is split into two
different groups. One group is for the Mental Health Overlay Services (MHOS) youth and the
other is for the Substance Abuse Overlay Services (SAOS) youth. All youth assigned to the
MHOS section of the program are required to complete the primary service. The program
conducts MRT on Mondays, Wednesdays, and Fridays for the youth assigned to the MHOS side
of the program. The program maintains the attendance sheets and monthly internal fidelity checks
in the SPEP binder titled MRT.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Moral Reconation Therapy was observed on 2/23/2022 via Zoom. The group began
at 3:35p and was held in one of the program’s classrooms. The facilitator was knowledgeable of
the material and followed the lesson plan. The facilitator began by going over the group rules. The
facilitator then had the youth explain what the group is about to a new youth. There were several
youth who presented their step to the group. The group ended at 4:32p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for annual review period of February 1, 2021 through January 31,
2022, the program has experienced facilitator turnover. One therapist left in October 2021 and the
clinical director left in November 2021. The therapist who left was facilitating a Moral Reconation
Therapy (MRT) group. The program transitioned the youth to another trained facilitator without
any gaps in service. There were no more than ten youth in a group per session, as indicated by
the attendance sheets. MRT is running year-round as an open group.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services. The director of case management stated the program will pull a co-
facilitator from one cohort in order to cover another cohort if a facilitator leaves the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were eleven fidelity checks
completed. The director of clinical services stated the program could not locate the monthly
internal fidelity checks for the months of January 2021, February 2021, and March 2021. The
director of clinical services also stated the program did complete monthly internal fidelity checks in
April 2021, but did not provide documentation. The program did not complete any internal fidelity
checks for the months of May 2021 and November 2021. The program was missing monthly
fidelity checks for the month of June 2021 for two therapists and the assistant clinical director;
July 2021 for two therapists and the assistant clinical director; July 2021 for one therapist; August
2021 for the former clinical director and three therapists; September 2021 for two therapists;
October 2021 for three therapists and the assistant clinical director; December 2021 for one
therapist and the assistant clinical director; and January 2022 for two therapists and the assistant
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
clinical director. The former clinical director and the current clinical director conducted the fidelity
checks during the annual review period.
According to the director of clinical services, the director of case management or another trained
facilitator conducts the monthly internal fidelity checks for the primary service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions noted for facilitators of Moral
Reconation Therapy during the annual review period.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period showed the evaluations have an area to
document if the facilitator has been trained in all facets of the position. The program provided
annual evaluations for three therapists and a ninety-day evaluation for one therapist. The annual
evaluations had a numerical value assigned for the performance of the facilitator as well as
detailed comments relating to the performance of that facilitator conducting Moral Reconation
Therapy. The ninety-day evaluation did not mention if the therapist facilitated any primary
services, nor did it have any detailed information regarding any facilitation.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 15 weeks for this type of service. Of the 40 youth in the
sample, 100% (40 of 40) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 45 contact hours for this type of service. Of the 40 youth in the
sample, 98% (39 of 40) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 20 youth
High = 9 youth
Total Youth in Sample = 40 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 23%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 9 youth
Total Youth in Sample = 40 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (34 of 40) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 23% (9 of 40) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 15 25 60%
Totals 78 100 78%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at Center for
Success and Independence - Ocala.
75 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 40 total youth sampled, all
40 included dosage with end dates in the EBS Module. Of those youth, 40 received at least the
recommended weeks of service. Youth in the sample completed between 18 and 53 weeks of
service, with an average of 29 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 40 total youth
sampled, all 40 included dosage in the EBS Module. Of those youth, 39 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 37 and 149 hours of
service, with an average of 80 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups and fidelity monitoring is
completed every month for each facilitator.
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 CSI-Ocala MRT.pdf


state: FL

summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence - Ocala
Youth Opportunity Investments, LLC.
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Period: February 1, 2021-January 31, 2022
SPEP Review Conducted: March 1, 2022
Florida Department of Juvenile Justice
Report Date: 5/23/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of February 1, 2021 through January 31,
2022 shows the program utilizes Impact of Crime (IOC) as their primary service. Impact of Crime
training is a three-day, 24-hour training currently being provided online via Microsoft Teams. The
documentation showed there were eleven staff members trained in the primary service. The
trained staff are six case managers, two shift supervisors, two transition service managers, and
the director of case management. All trained staff, except the director of case management, have
facilitated IOC during the annual review period. Any trained staff can conduct the internal fidelity
checks. The director of case management, a transition service manager, a case manager, and a
shift supervisor conducted the monthly internal fidelity checks for the annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all seven
chapters of the service and provides a script for the facilitator to follow. The manual also has an
appendix and a student guide with all the student exercises. The program makes copies of the
student exercises and passes them out to the youth during each lesson. The program conducts
IOC on Tuesdays and Thursdays. The director of case management indicates the program will
conduct IOC on Mondays, Wednesdays, or Fridays as needed. The program keeps all SPEP
documentation (attendance sheets, fidelity checks, and training certificates) in a binder labeled
IOC SPEP documentation. All youth admitted to the program are required to participate in the
primary service. The program is split into two different groups. One group is for the Mental Health
Overlay Services (MHOS) youth and the other is for the Substance Abuse Overlay Services
(SAOS) youth. IOC is a closed group with no more than twelve youth assigned to one cohort.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 2/22/2022 via Zoom. The group began at 4:30p
and was held in one of the program’s classrooms. The facilitator was knowledgeable of the
material and followed the lesson plan. One youth said the program is refusing to allow him to
make an abuse call. The technical assistance specialist emailed the director of case management
to inform her what the youth stated. The director of case management stated she would speak
with the youth and ask if he would like to make his abuse call. The other youth were holding side
conversations and being disrespectful to the facilitator. The facilitator did her best to redirect the
youth. The group ended at 5:30p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation for the annual review period of February 1, 2021 through January 31,
2022, the program has experienced facilitator turnover which adversely affected the delivery of
Impact of Crime (IOC). Two case managers and one transition service manager left the program
while they were in the middle of a cohort. Another trained facilitator took over the cohort without
any break in service. There were no gaps in service during the annual review period.
There were fourteen cohorts run during the review period. The program splits the youth into two
sections: Mental Health Overlay Services (MHOS) and Substance Abuse Overlay Services
(SAOS).
Seven of the fourteen cohorts were facilitated on the SAOS side.
The first cohort began on 2/23/2021 with nine youth and ended with four youth. Four of the youth
were released and one youth was arrested. One of the case managers began the cohort, but left
the program on 3/11/2021. Another trained facilitator and co-facilitator took over the group. The
co-facilitator also left the program on 4/28/2021.
The second cohort began on 4/28/2021 with ten youth and ended with ten youth. A transition
service manager was added as a co-facilitator on 5/12/2021 and removed on 6/25/2021.
The third cohort began on 4/28/2021 with only one youth. The director of case management
stated the youth is by himself for safety and security reasons. The director of case management
also stated the program is trying to reintegrate the youth back into another cohort, but it could not
be done safely.
The fourth cohort began on 6/15/2021 with five youth and ended with six youth. A case manager
co-facilitated up to 6/18/2021. There was no co-facilitator during the rest of the cohort. One youth
was added to the cohort on 6/22/2021. The facilitator conducted individual make up sessions to
ensure the youth was caught up with the cohort. Another youth was added to the cohort on
8/27/2021. The youth was transferred from another program and was in an Impact of Crime
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
cohort at the prior program. The facilitator made sure the youth was caught up with the cohort.
One youth was transferred on 9/10/2021 and one youth was released on 10/14/2021.
The fifth cohort began on 7/27/2021 with five youth and ended with five youth. The shift supervisor
facilitated the cohort up to 11/15/2021. A case manager took over the cohort for the last two
sessions on 11/18/2021 and 11/21/2021.
The sixth cohort began on 11/9/2021 began with seven youth and currently has seven youth. The
seventh cohort began on 1/4/2022 with eight youth and currently has eight youth.
The remaining seven of the fourteen cohorts were conducted on the MHOS side.
The eighth cohort began on 2/23/2021 with eight youth and ended with six youth. Two of the
youth were released on 5/21/2021.
The ninth cohort began on 2/23/2021 with eleven youth and ended with one youth. Ten of the
youth were released prior to completing the primary service.
The tenth cohort began on 4/28/2021 with eight youth and ended with seven youth. One youth left
the program on 6/18/2021.
The eleventh cohort began on 9/21/2021 with nine youth and ended with seven youth. One youth
was released on 10/1/2021. One youth ended on 12/9/2021; the youth was at court from
12/13/2021-1/5/2022.
The twelfth cohort began on 6/22/2021 with seven youth and ended with seven youth. A transition
service manager began the cohort as a co-facilitator and was replaced by a shift supervisor on
6/29/2021. The shift supervisor stopped co-facilitating after 7/29/2021.
The thirteenth cohort began on 11/9/2021 with nine youth and currently has seven youth. One
youth left the program on 11/11/2021 and another youth left on 12/7/2021.
The fourteenth cohort began on 1/4/2022 with eight youth and currently has eight youth. The sign
in sheets provided did not list a facilitator for 1/25/2022, 1/22/2022, 1/20/2022, 1/15/2022,
1/12/2022, 1/6/2022, and 1/4/2022.
According to the director of clinical services, Youth Opportunity Investments provides staff
incentives to decrease the likelihood of turnover in implementation of the Standardized Program
Evaluation Protocol services. The director of case management stated the program will pull a co-
facilitator from one cohort in order to cover another cohort if a facilitator leaves the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period showed there were forty-one fidelity
checks completed for facilitators of Impact of Crime. There were no internal fidelity checks
completed for February 2021. There were missing fidelity checks for April 2021 for a case
manager; November 2021 for a shift supervisor; June 2021 for a transition service manager;
December 2021 for a transition service manager; and a case manager for January 2022. There
was a fidelity check completed on 5/28/2021, but the observer was also listed as the co-facilitator
for the group.
The director of case management stated the transition service manager left the program during
the month of December 2021 prior to having a fidelity check completed. The director of case
management was unable to complete a fidelity check for the month of June 2021. The November
2021 fidelity for the shift supervisor was scheduled in advance; however, the shift supervisor
experienced an unforeseen event which caused him to miss work for some weeks. As a result,
another facilitator completed the last few groups.
According to the director of clinical services, the director of case management or another trained
facilitator conducts the monthly internal fidelity checks for the primary service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation showed there were no corrective actions for facilitators of Impact of
Crime noted during the annual review period.
According to the director of clinical services, all facilitators who do not provide adequate
facilitation of Standardized Program Evaluation Protocol services will receive progressive
discipline to include retraining, coaching, and formal written write-up.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period showed the evaluations have an area to
document if the facilitator has been trained in all facets of the position. The program provided
evaluations for four case managers and one transition service manager. The evaluations had a
numerical value assigned for the performance of the facilitator as well as detailed comments
relating to the performance of that facilitator conducting Impact of Crime.
According to the director of clinical services, all facilitators are evaluated yearly on group services
provided. Evaluations include implementation and youth engagement.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 70 youth in the
sample, 74% (52 of 70) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 70 youth in the
sample, 79% (55 of 70) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 10 youth
Moderate-High = 30 youth
High = 23 youth
Total Youth in Sample = 70 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 23 youth
Total Youth in Sample = 70 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (63 of 70) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 33% (23 of 70) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 72 85 85%
This SPEP report evaluates Impact of Crime, an intervention delivered at Center for Success and
Independence - Ocala.
75 youth completed the program during the review period. Of those youth, 70 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 70 total youth sampled, only
69 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 52
received at least the recommended weeks of service. Youth in the sample completed between 3
and 19 weeks of service, with an average of 15 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program earned 6 points for Amount of Service: Contact Hours. Of the 70 total youth
sampled, only 69 included dosage in the EBS Module. Of those youth with correct dosage, 55
received at least the recommended hours of service. Youth in the sample completed between 6
and 36 hours of service, with an average of 29 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center for Success and Independence - Ocala can optimize their SPEP Quality of Service
Delivery score by ensuring facilitator turnover does not impact groups and fidelity monitoring is
completed every month for each facilitator
Center for Success and Independence - Ocala can optimize their SPEP Amount of Service score
by ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

source: DJJ

report_type: SPEP

file_name: FY2021-22 CSI-Ocala IOC.pdf

program_status_note: Open

service_type: Impact of Crime