state: FL

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center of Success and Independence-Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Impact of Crime (MHOS)
SPEP Review Period: 2/1/2023-1/31/2024
SPEP Review Conducted: 2/6/2024-2/7/2024
Florida Department of Juvenile Justice
Report Date: 6/21/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation reviewed for 2/1/2023 through 1/31/2024 shows the program uses Impact of
Crime as a primary service. Facilitator training consists of a three day twenty-four (24) hour
training provided by the Department of Juvenile Justice. This training is available to new
facilitators both in-person and via Microsoft Teams. The documentation showed there were
thirteen (13) staff members trained in the primary service: one state director of clinical services,
one director of case management, one facility administrator, one transitional service manager,
and nine case managers. The director of case management and six case managers facilitated
Impact of Crime for the Mental Health Overlay Services (MHOS) population.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual titled Impact of Crime: Addressing the Harm. The manual includes
seven chapters and provides a script for the facilitator to follow. The curriculum includes a student
guide with exercises for the participants to complete that are aimed at reinforcing the material
presented during group sessions. The program does not assign a student guide to each individual
participant, rather the facilitator makes copies of the exercises and provides one to each youth
during the lesson. The director of case management indicated the program conducts Impact of
Crime on Tuesdays and Thursdays. The program maintains all SPEP documentation (attendance
sheets and internal fidelity checks) in a separate binder labeled “Impact of Crime (MHOS).” All
youth admitted to the program are required to participate in the primary service. Impact of Crime
is a closed group with no more than twelve (12) youth assigned to a cohort.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on-site on 11/7/2023. The group began at 10:55 a.m.
in a classroom. A copy of the lesson plan was not provided. There were twelve (12) youth, one
facilitator, one direct care staff, and the facility administrator present. The group began with an ice
breaker activity and a review of the previous lesson before proceeding to the lesson of the day.
The facilitator did have a copy of the facilitator’s manual available during this session. The group
engaged in a discussion about being aware of consequences a person can suffer if they give in to
their triggers. No behavior management issues were observed. The group ended at 12:10 p.m.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 showed
the program has experienced facilitator turnover. This turnover did adversely affect the delivery of
Impact of Crime, as one case manager left during the middle of a cohort. Four case managers left
the program. There were no gaps in service during the annual review period caused by a lack of
trained facilitators. There were six cohorts of Impact of Crime completed and two cohorts are
ongoing for the MHOS population.
The first cohort began on 10/18/2022 with twelve (12) youth and ended on 2/20/2023 with ten(10)
youth.
The second cohort began on 12/13/2022 with nine youth and ended on 4/4/2023 with eight youth.
One youth was successfully discharged prior to the end of the cohort.
The third cohort began on 2/21/2023 with seven youth and ended on 6/8/2023 with five youth.
Two youth were dropped from this cohort. One youth was transferred out of the program and one
youth was arrested prior to the end of the cohort.
The fourth cohort began on 4/18/2023 with eight youth and ended on 8/3/2023 with eight youth.
The fifth cohort began on 6/13/2023 with nine youth and ended on 9/28/2023 with nine youth.
The sixth cohort began on 8/29/2023 with twelve (12) youth and ended on 12/14/2023 with twelve
(12) youth.
The seventh cohort began on 10/31/2023 with twelve (12) youth and was ongoing at the time of
this review with twelve (12) youth.
The eighth cohort began on 1/16/2024 with twelve (12) youth and was ongoing with twelve (12)
youth.
An informal interview with the director of case management confirmed the program will replace
one facilitator with another qualified facilitator should there be any staff turnover in the middle of a
cohort.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 showed
twenty-six (26) out of the required twenty-eight (28) internal fidelity monitoring checks were
completed. According to the documentation, either the director of case management or another
trained facilitator conducted the monthly internal fidelity monitoring checks for this primary service
during this review period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 showed
there were no corrective actions identified during the internal fidelity checks. The director of case
management explained during an informal interview if a need for corrective action is found, the
facilitator will receive progressive discipline to include coaching, retraining, and formal write-up(s).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Copies of the annual employee evaluations were not provided to the technical assistance
specialist for the annual review period of 2/1/2023 through 1/31/2024.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 104 youth in
the sample, 97% (101 of 104) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 104 youth in the
sample, 100% (104 of 104) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 82%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 19 youth
Moderate-High = 32 youth
High = 34 youth
Total Youth in Sample = 104 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 34 youth
Total Youth in Sample = 104 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 82% (85 of 104) youth scored Moderate to High-Risk to reoffend, for a
score of 7 points.
Of the SPEP sample, 33% (34 of 104) youth scored High-Risk to reoffend, for a score of 10
points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 17 25 68%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Center of Success and
Independence-Ocala.
105 youth completed the program during the review period. Of those youth, 104 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 104 total youth sampled, all
104 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 101
received at least the recommended weeks of service. Youth in the sample completed between 13
and 20 weeks of service, with an average of 17 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 104 total youth
sampled, all 104 included dosage in the EBS Module. Of those youth with correct dosage, 104
received at least the recommended hours of service. Youth in the sample completed between 26
and 68 hours of service, with an average of 34 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center of Success and Independence-Ocala can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups, fidelity monitoring is completed
every month for each facilitator, and each facilitator’s performance evaluation includes discussion
of their skill in delivering this primary service.
Center of Success and Independence-Ocala can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2023-24 CSI OCALA IOC - MHOS.pdf

program_name_raw: Center for Success and Independence - Ocala

program_status_note: Open

service_type: Impact of Crime (MHOS)


facility_name: Center for Success and Independence - Ocala

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence-Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Impact of Crime (SAOS)
SPEP Review Period: 2/1/2023-1/31/2024
SPEP Review Conducted: 2/6/2024-2/7/2024
Florida Department of Juvenile Justice
Report Date: 6/21/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 shows
the program uses Impact of Crime as a primary service. Facilitator training consists of a three-
day, twenty-four (24) hour training provided by the Department of Juvenile Justice. This training is
available to new facilitators both in-person and via Microsoft Teams. The documentation shows
there were thirteen (13) staff members trained in the primary service: one state director of clinical
services, one director of case management, one facility administrator, one transitional service
manager, and nine case managers. The director of case management and six case managers
facilitated Impact of Crime for the Substance Abuse Overlay Services (SAOS) population.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual titled Impact of Crime: Addressing the Harm. The manual includes
seven chapters and provides a script for the facilitator to follow. The curriculum includes a student
guide with exercises for the participants to complete. The program does not assign a student
guide to each participant, rather the facilitator makes copies of the exercises. The program holds
Impact of Crime on Tuesdays and Thursdays. The program maintains all SPEP documentation
(attendance sheets and internal fidelity checks) in a separate binder labeled "Impact of Crime
(SAOS)." All youth admitted to the program are required to participate in this primary service.
Impact of Crime is a closed group with no more than twelve (12) youth assigned to a cohort.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
A session of Impact of Crime was observed on-site on 1/20/2023. The group began at 10:55 a.m.
in a classroom. A copy of the lesson plan was not provided to the technical assistance specialist
for review. There were nine youth, one facilitator, and the facility administrator present. The group
began with an ice breaker activity and a brief review of the previous lesson before proceeding to
the lesson of the day. The facilitator did have a copy of the facilitator’s manual available during
this session. The group engaged in a discussion about the impact of crime on both the offender
and the victim. No behavior management issues were observed. The group ended at 12:00 p.m.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 showed
the program has experienced facilitator turnover. This turnover did adversely affect the delivery of
Impact of Crime. Four case managers left the program in during the annual review period,
including one who left during the middle of a cohort. There were no gaps in service during the
annual review period caused by a lack of trained facilitators. There were five cohorts completed
and two cohorts are ongoing for the SAOS population at the time of this annual review.
The first cohort began on 12/14/2022 with ten (10) youth and ended on 4/4/2023 with ten (10)
youth.
The second cohort began on 2/7/2023 with eleven (11) youth and ended on 6/1/2023 with eleven
(11) youth.
The third cohort began on 5/9/2023 with seven youth and ended on 8/29/2023 with seven youth.
The fourth cohort began on 7/18/2023 with nine youth and ended on 11/2/2023 with eight youth.
One youth was successfully released from the program during this cohort.
The fifth cohort began on 8/22/2023 with eleven (11) youth and ended on 12/14/2023 with eleven
(11) youth.
The sixth cohort began on 10/31/2023 with twelve (12) youth and was ongoing at the time of this
annual review with twelve (12) youth.
The seventh cohort began on 1/16/2024 with ten (10) youth and was ongoing at the time of this
annual review with ten (10) youth.
An informal interview with the director of case management confirmed the program will replace
one facilitator with another qualified facilitator should there be any staff turnover in the middle of a
cohort.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of 2/1/2023 through 1/31/2024 showed
there were twenty-five (25) out of the required twenty-five (25) internal fidelity monitoring checks
completed for the facilitators of Impact of Crime.
According to the reviewed documentation, either the director of case management or another
trained facilitator conducts the monthly internal fidelity monitoring checks for this primary service.
Any facilitator that is trained in Impact of Crime can conduct the internal fidelity monitoring.
Internal fidelity monitoring checks during the annual review period were conducted by the director
of case management, one transitional service manager, one facility administrator, and five case
managers who were trained in the service.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for the annual review period of 2/1/2023 through 1/31/2024 shows
there were no corrective actions identified. The director of case management explained during an
informal interview if a need for corrective action is found, the facilitator will receive progressive
discipline to include coaching, retraining, and formal write-up(s).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Copies of the annual employee evaluations were not provided to the technical assistance
specialist for the annual review period of 2/1/2023 through 1/31/2024.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 104 youth in
the sample, 97% (101 of 104) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 104 youth in the
sample, 100% (104 of 104) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 82%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 19 youth
Moderate-High = 32 youth
High = 34 youth
Total Youth in Sample = 104 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 34 youth
Total Youth in Sample = 104 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 82% (85 of 104) youth scored Moderate to High-Risk to reoffend, for a
score of 7 points.
Of the SPEP sample, 33% (34 of 104) youth scored High-Risk to reoffend, for a score of 10
points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 17 25 68%
Totals 75 85 88%
This SPEP report evaluates Impact of Crime, an intervention delivered at Center of Success and
Independence-Ocala.
105 youth completed the program during the review period. Of those youth, 104 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 104 total youth sampled, all
104 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 101
received at least the recommended weeks of service. Youth in the sample completed between 13
and 20 weeks of service, with an average of 17 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 104 total youth
sampled, all 104 included dosage in the EBS Module. Of those youth with correct dosage, 104
received at least the recommended hours of service. Youth in the sample completed between 26
and 68 hours of service, with an average of 34 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 98%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center of Success and Independence-Ocala can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Center of Success and Independence-Ocala can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 CSI OCALA IOC - SAOS.pdf

service_type: Impact of Crime (SAOS)


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence-Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Moral Reconation Therapy
SPEP Review Period: 2/1/2023-1/31/2024
SPEP Review Conducted: 2/6/2024-2/7/2024
Florida Department of Juvenile Justice
Report Date: 6/21/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of the documentation for the annual review period of February 1, 2023 through January
31, 2024 shows the program uses Moral Reconation Therapy (MRT) as a primary service for the
Mental Health Overlay Services (MHOS) population. MRT groups are conducted by therapists
that have been formally trained in the service. MRT training is conducted by a qualified trainer
either in-person or virtually. The documentation shows there were eight staff members trained to
facilitate MRT during the review period: one clinical director, one designated mental health
clinician authority, and six therapists. MRT was facilitated by the clinical director and six therapists
during the annual review period. The clinical director and the designated mental health clinician
authority conducted the monthly internal fidelity monitoring for this service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
There is a formal facilitator’s manual and separate student manual that is distributed to each
youth. Moral Reconation Therapy (MRT) is designed to be an open group and group sessions are
held year-round. This curriculum is designed to help juvenile offenders to reexamine their choices
and develop appropriate decision-making skills. Each youth who enters the MHOS population is
immediately enrolled in the service based on the youth’s assigned living module. The program
maintains the attendance sheets in a SPEP binder labeled “Moral Reconation Therapy”.
Documentation of internal fidelity monitoring is maintained in a separate binder labeled “Fidelity
Monitoring.” MRT groups are held three times weekly on Tuesdays, Wednesdays, and Thursdays
at 2:45 p.m.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of a Moral Reconation Therapy group took place on-site on September 20, 2023
at 2:45 p.m. The group consisted of one facilitator, nine youth, and one direct care staff. The
group was held in a classroom. The facilitator used the facilitator’s manual throughout this group
session and was knowledgeable about the material being presented to the group. The facilitator
used his own words to convey the information. This session began with a check-in, followed by an
ice breaker and a review of the group rules. The group actively participated in a discussion about
various rules that the youth might choose to give to their children in the future. The youth debated
why each rule would be important and whether the rule would be a realistic expectation. The
facilitator utilized effective motivational interviewing skills to encourage the youth to share more
about their personal experiences and to assess for understanding. There were no behavioral
issues observed during this group. The group ended at 3:45 p.m.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation for the annual review period of February 1, 2023 through January
31, 2024 shows there was facilitator turnover. One therapist left the program in November 2023.
This turnover did adversely affect the delivery of this primary service. The affected group was
reassigned to a new facilitator immediately. There were no gaps in service during the annual
review period. An informal interview with the clinical director confirmed if a facilitator should leave
the program, another trained therapist will then take over the group. The program provided
documentation of four on-going cohorts for this service during the review period.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the annual review period of February 1, 2023 through January
31, 2024 shows a total of fifty-one (51) out of fifty-four (54) required monthly fidelity checks were
conducted for Moral Reconation Therapy (MRT). The clinical director and the designated mental
health clinician authority conducted the internal fidelity monitoring and have been formally trained
in MRT.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for the annual review period of February 1, 2023 through January
31, 2024 shows there were no corrective actions identified during the internal fidelity monitorings.
The program has a system in place to address the need for corrective actions identified during
monthly internal fidelity monitoring of Moral Reconation Therapy.
According to the clinical director, facilitators in need of corrective action related to their delivery of
the service will receive progressive discipline in the form of coaching, additional training, and
formal write-ups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Copies of the annual employee evaluations and/or ninety (90) day reviews were not provided to
the technical assistance specialist for this annual review period of February 1, 2023 through
January 31, 2024.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 53 youth in the
sample, 98% (52 of 53) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 53 youth in the
sample, 9% (5 of 53) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 10 youth
Moderate-High = 17 youth
High = 17 youth
Total Youth in Sample = 53 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 32%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 17 youth
Total Youth in Sample = 53 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (44 of 53) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 32% (17 of 53) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 17 25 68%
Totals 70 100 70%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at Center of
Success and Independence-Ocala.
105 youth completed the program during the review period. Of those youth, 53 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 53 total youth sampled, only
52 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 52
received at least the recommended weeks of service. Youth in the sample completed between 25
and 62 weeks of service, with an average of 33 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 53 total youth
sampled, all 53 included dosage in the EBS Module. Of those youth with correct dosage, 5
received at least the recommended hours of service. Youth in the sample completed between 16
and 50 hours of service, with an average of 32 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 78%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center of Success and Independence-Ocala can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups, fidelity monitoring is completed
every month for each facilitator, and each facilitator’s performance evaluation includes discussion
of their skill in delivering this primary service.
Center of Success and Independence-Ocala can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 CSI OCALA MRT.pdf

service_type: Moral Reconation Therapy


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Center for Success and Independence-Ocala
Youth Opportunity Investments, LLC
(Contract Provider)
4055 NW 105th Street
Ocala, Florida 34482
Primary Service: Seven Challenges
SPEP Review Period: 2/1/23-1/31/24
SPEP Review Conducted: 2/6/24-2/7/24
Florida Department of Juvenile Justice
Report Date: March 18, 2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of February 1, 2023 through January 31, 2024
shows the program uses Seven Challenges as a primary service for the Substance Abuse
Overlay Services (SAOS) population. Seven Challenges groups are conducted by therapists that
have been formally trained in the service. The training is conducted by a qualified trainer either in-
person or virtually. The documentation shows there were eight staff members trained to facilitate
Seven Challenges during the review period: one clinical director, one designated mental health
clinician authority, and six therapists. Seven Challenges was facilitated by one clinical director and
five therapists during the annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of February 1, 2023 through January
31, 2024 shows the program uses the Seven Challenges Manual, Second Edition. The manual
includes both activities for the youth to complete in group and instructions to aid facilitators as
they present to the group. Seven Challenges is designed to be an open group and group sessions
are held year-round. Every SAOS youth who enters the program is immediately enrolled in the
service and is required to complete this primary service prior to discharge. The program maintains
the attendance sheets in a SPEP binder labeled “Seven Challenges.” Documentation of internal
fidelity monitoring is maintained in a separate binder labeled “Fidelity Monitoring.” Seven
Challenges groups are currently held weekly on Tuesdays, Wednesdays, and Thursdays at 10:55
a.m. and 4:00 p.m.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Seven Challenges took place on-site on February 7, 2024. The group consisted
of one facilitator, nine youth, three direct care staff, and the clinical director. The group was held in
a classroom. The facilitator used the manual throughout this group session and was
knowledgeable about the material being presented to the group. The facilitator used her own
words to convey the information. This session began with an ice breaker, followed by a review of
the previous lesson. The facilitator utilized effective motivational interviewing skills to encourage
the youth to share more about their personal experiences and to assess for understanding. There
were no behavioral issues observed during this group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation for Seven Challenges for the annual review period of February 1,
2023 through January 31, 2024 shows there was facilitator turnover. One therapist left the
program in November 2023. This turnover did adversely affect the delivery of this primary service
as the therapist was facilitating a Seven Challenges group at the time of their departure. The
affected group was reassigned to a new facilitator immediately. There were no gaps in service
during this annual review period due to a lack of trained facilitators. An informal interview with the
clinical director confirmed if a facilitator should leave the program, another trained therapist will
then take over the group. The program provided documentation of five on-going cohorts for this
service during the review period.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for Seven Challenges for the annual review period of February 1,
2023 through January 31, 2024 shows a total of fifty (50) out of fifty-one (51) required fidelity
checks were completed. The internal fidelity monitoring was conducted by the clinical director and
the designated mental health clinician authority and both were formally trained in this primary
service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
f. Corrective Action based on Fidelity Monitoring Basic Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for Seven Challenges for the annual review period of February 1,
2023 through January 31, 2024 shows there were no corrective actions identified during the
internal fidelity monitorings. The program has a system in place to address the need for corrective
actions identified during monthly internal fidelity monitoring of Seven Challenges. According to the
clinical director, facilitators in need of corrective action related to their delivery of the service will
receive progressive discipline in the form of coaching, additional training, and formal write-ups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Copies of the annual employee evaluations and/or ninety (90) day reviews were not provided to
the technical assistance specialist for this annual review period of February 1, 2023 through
January 31, 2024.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 50 youth in the
sample, 94% (47 of 50) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 10%
Research indicates a target of 40 contact hours for this type of service. Of the 50 youth in the
sample, 100% (50 of 50) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 9 youth
Moderate-High = 15 youth
High = 16 youth
Total Youth in Sample = 50 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 32%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 16 youth
Total Youth in Sample = 50 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (40 of 50) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 32% (16 of 50) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 17 25 68%
Totals 75 95 79%
This SPEP report evaluates Seven Challenges, an intervention delivered at Center for Success
and Independence-Ocala.
105 youth completed the program during the review period. Of those youth, 50 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 50 total youth sampled, only
49 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 47
received at least the recommended weeks of service. Youth in the sample completed between 21
and 53 weeks of service, with an average of 32 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 50 total youth
sampled, all 50 included dosage in the EBS Module. Of those youth with correct dosage, 50
received at least the recommended hours of service. Youth in the sample completed between 70
and 85 hours of service, with an average of 72 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 87%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Center for Success and Independence-Ocala can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups, fidelity monitoring is completed
every month for each facilitator, and each facilitator’s performance evaluation includes discussion
of their skill in delivering this primary service.
Center for Success and Independence-Ocala can optimize their SPEP Amount of Service score
by ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the
full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 CSI OCALA Seven Challenges.pdf

service_type: Seven Challenges