summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Thinking For a Change
SPEP Review Date(s): June 4, 2018
Florida Department of Juvenile Justice
Report Date(s): October 5, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Marion Youth Academy MQI Program Code: 8304
Provider Name: Sequel Youth and Family Services Contract Number: 10321
Location: Marion County / Circuit: 5 Number of Beds: 48
Review Date(s): June 4, 2018 Lead Reviewer Code: 157
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 6 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation reviewed reflected six staff memebers are trained to provide this intervention. The
program has a thirty-two hour training protocol for this intervention. In the past twelve months four
staff members have provided the intervention.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Thinking For A Change Version 4.0 manual. The manual includes twenty-
five lessons which detail summary and rationale, concepts and definitions, objectives, major
activities, supplements, and special notes. The manual further specifies the frequency and
duration of the service.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This intervention was not observed during this review.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
There has been no gap in service delivery of this intervention and no turnover of facilitator. Four
case managers provided the intervention over the past twelve months.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Twelve months of internal fidelity monitoring reports were reviewed. The program has an internal
process to perform internal fidelity monitoring. During the past twelve months thirty instances of
fidelity monitoring were required and each was completed.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective action based on internal fidelity monitoring. Each fidelity
monitoring checklist includes a section for feedback and recommendations for facilitator
improvement. During the last twelve months, there were no corrective actions needed
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Facilitators for this intervention received performance evaluations which included specific
information about their delivery of the intervention.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 7 youth in the
sample, 86% (6 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 7 youth in the
sample, 43% (3 of 7) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 4 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (7 of 7) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 57% (4 of 7) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 25 25 100%
Totals 92 100 92%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Marion Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 7 total youth sampled, only
6 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 6
received at least the recommended weeks of service. Youth in the sample completed between 19
and 23 weeks of service, with an average of 21 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 7 total youth sampled,
all 7 included dosage in the EBS Module. Of those youth, 3 received at least the recommended
hours of service. Youth in the sample completed between 37 and 46 hours of service, with an
average of 40 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


state: FL

facility_name: Marion Regional Juvenile Detention Center

summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Anger Management for Substance Abuse and Mental
Health Clients
SPEP Review Date(s): 3/26/19-3/29/19
Florida Department of Juvenile Justice
Report Date(s): 5/3/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Substance Abuse and Mental
Health Clients.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the twelve-month review period showed there were six staff trained
to facilitate Anger Management for Substance Abuse and Mental Health Clients. There are two
staff who were trained but no longer work at the program. Anger Management training is an in-
service training conducted by the clinical director. The documentation reviewed showed there
were three staff conducting cohorts during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Anger Management for Substance Abuse and Mental Health Clients
manual. The manual was provided for review. The manual includes twelve sessions. Each lesson
has a script for the facilitator and instructions for each activity. The manual requires that the
lessons be given in order. The manual provides specific instructions on the implementation of the
program. There are no separate student workbooks. The facilitators make copies of the current
lessons to hand out to the youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The facilitator went over a review of last week. The facilitator asked the youth to tell him what
happened when they got angry this week. The facilitator then asked the youth what they did, how
they coped with it, and how they de-escalated the situation. There were three youth who had their
heads on the table, not participating in group. One youth had behavioral issues. The facilitator
went around the group asking the youth to state what they learned during the session. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
facilitator then had the youths break up into smaller groups. The facilitator had the youth act out
scenarios. The youth had to tell a physical cue and then explain the coping skill used. The
facilitator ended the session with a review of the lesson.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the twelve-month review period showed there was facilitator
turnover at the program. The turnover did not affect the primary service. Cohorts were run
consistently throughout the review period. The clinical director stated the program's policy for
turnover is to ensure everyone is trained in the curriculum. If there is facilitator turnover, then
another trained facilitator will take over the group. There was documentation of the program
completing five cohorts during the review period. There are currently two cohorts being run during
the time of the review.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director stated the program's policy for fidelity monitoring is to have the clinical
director complete monthly fidelity monitoring for the mental health services. Fidelity monitoring
was completed each month for each facilitator, without exception, during the review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the clinical director, the program's policy for corrective actions noted on the fidelity
monitoring is to have the clinical director sit down with the therapist to discuss the feedback. The
clinical director will provide written strategies and goals with the facilitator. The clinical director will
go over the strengths and weaknesses. The clinical director will then create two to three goals or
strategies for the facilitator. The clinical director will increase the fidelity monitoring to weekly until
the corrective action is corrected. The documentation reviewed during the review period showed
no corrective actions noted nor taken.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program had four staff evaluations available for review.
The evaluations were completed on the case managers, transition specialist, and director of case
management. All evaluations have an area that addresses the facilitator's skill in the primary
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 76 youth in the
sample, 0% (0 of 76) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 76 youth in the
sample, 0% (0 of 76) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 28 youth
High = 32 youth
Total Youth in Sample = 76 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 32 youth
Total Youth in Sample = 76 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (69 of 76) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (32 of 76) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 73 95 77%
This SPEP report evaluates Anger Management for Substance Abuse and Mental Health Clients,
an intervention delivered at Marion Youth Academy.
76 youth completed the program during the review period. Of those youth, all were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 76 total youth sampled, only
71 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 11
and 13 weeks of service, with an average of 11 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 76 total youth
sampled, only 73 included dosage in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended hours of service. Youth in the sample completed between 21
hours of service.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Marion Youth Academy

program_status_note: Closed


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Date(s): 3/26/19‐3/29/19
Florida Department of Juvenile Justice
Report Date(s): 5/3/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the twelve-month review period showed there were three staff
trained to facilitate Impact of Crime. Impact of Crime training is a four day training provided by a
qualified trainer. The documentation reviewed showed each of the three staff facilitated groups
during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime Addressing the Harm manual. The manual was provided
for review. The manual includes seven sections broken down into lessons. Each lesson has a
script for the facilitator and instructions for each activity. The manual requires that the lessons be
given in order and provides specific instructions on the implementation of the program. There are
no separate student workbooks. The facilitators make copies of the current lessons to hand out to
the youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The facilitator has the lesson outline written down on the board for the youth to preview. The
facilitator explained the outline of the lesson to the youth, conducted an icebreaker, then reviewed
the previous lesson. The facilitator played a video for the youth to watch and had the youth look
for the victim impact and victim's rights in the video. The facilitator then went over the video and
discussed victim impact. The facilitator passed out a handout and asked the youth to answer the
questions on the handout.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the review period showed there was facilitator turnover at the
program; however, the turnover did not affect the primary service. The cohorts were run
continuously without gaps in service. Documentation indicated the program completed three
cohorts during the review period. A fourth cohort just began the week of the review. The clinical
director stated the program's policy for turnover is to ensure everyone is trained in the SPEP
curriculum. If there is facilitator turnover, then another trained facilitator will take over the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Fidelity monitoring was completed each month for each facilitator, without exception. The clinical
director stated the program's policy for fidelity monitoring is for the clinical director to complete
monthly fidelity monitoring for the mental health services. The director of case management or
other trained facilitator will conduct the fidelity monitoring for the other services.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Documentation reviewed showed no corrective actions were noted nor taken. According to the
clinical director, the program's policy for corrective actions noted on the fidelity monitoring is to
have the clinical director sit down with the therapist to discuss the feedback. The clinical director
will provide the facilitator written strategies and goals. The clinical director will go over the
strengths and weaknesses. The clinical director will increase the fidelity monitoring to weekly until
the corrective action is corrected.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the documentation showed the program had three staff evaluations available for
review. Evaluations were completed on the case managers, transition specialist, and director of
case management. All evaluations have an area that addresses the facilitator's skill in the primary
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 75 youth in the
sample, 49% (37 of 75) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 75 youth in the
sample, 53% (40 of 75) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 28 youth
High = 31 youth
Total Youth in Sample = 75 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 31 youth
Total Youth in Sample = 75 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (68 of 75) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 41% (31 of 75) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime, an intervention delivered at Marion Youth Academy.
76 youth completed the program during the review period. Of those youth, 75 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 75 total youth sampled, only
41 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 37
received at least the recommended weeks of service. Youth in the sample completed between 13
and 23 weeks of service, with an average of 19 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 75 total youth
sampled, only 41 included dosage in the EBS Module. Of those youth with correct dosage, 40
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
received at least the recommended hours of service. Youth in the sample completed between 23
and 34 hours of service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Thinking for a Change
SPEP Review Date(s): 3/26/19‐3/29/19
Florida Department of Juvenile Justice
Report Date(s): 5/3/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of the documentation for thereview period showed there are six staff trained to facilitate
Thinking for a Change. There are two transition service managers, three case managers, and the
director of case management trained. Thinking for a Change training is a thirty-two-hour training
completed by a qualified trainer. The documentation reviewed showed four staff facilitated groups
during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Thinking for a Change 4.0 manual. Technical Assistance Specialist
reviewed the manual. The manual includes twenty-five lessons. Each lesson has a script for the
facilitator and instructions for each activity. The manual requires that the lessons be given in
order. The manual provides specific instructions on the implementation of the program. There are
no separate student workbooks. The facilitators make copies of the current lessons to hand out to
the youth. There was documentation of the program completing four cohorts during the review
period as evidenced by the sign in sheets. There is currently one cohort being ran during the time
of the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The facilitator had the class rules and expectations, giving feedback, and the skill steps written on
chart paper for the youth to see. The facilitator began group with a review of the homework from
the prior session. The facilitator then modeled the skill "knowing your feelings". The facilitator
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
went over the scenario, then asked the youth to role play one of three scenarios. Youth were not
following instructions, talking with the staff and each other. The facilitator continued attempts to
get each group of two youth to figure out their role play. The facilitator followed the script for the
lesson. The facilitator assigned homework for the youth to complete.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the review period showed there was facilitator turnover at the
program; however, the turnover did not affect the primary service. The cohorts were run
continuously without gaps in service. Documentation indicated the program completed four
cohorts during the review period. A fifth cohort was ongoing during the week of the review. The
program's policy for turnover is to ensure everyone is trained in the SPEP curriculum. If there is
facilitator turnover, then another trained facilitator will take over the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Fidelity monitoring was completed each month for each facilitator, without exception. The clinical
director stated the program's policy for fidelity monitoring is for the clinical director to complete
monthly fidelity monitoring for the mental health services. The director of case management or
other trained facilitator will conduct the fidelity monitoring for the other services.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Documentation reviewed showed no corrective actions were noted nor taken. According to the
clinical director, the program's policy for corrective actions noted on the fidelity monitoring is to
have the clinical director sit down with the therapist to discuss the feedback. The clinical director
will provide the facilitator written strategies and goals and go over strengths and weaknesses. The
clinical director will increase the fidelity monitoring to weekly until the corrective action is closed.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program had six staff evaluations available for review. The
evaluations were completed on the case managers, transition specialist, and director of case
management. All evaluations have an area which addresses the facilitator's skill in the primary
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 73 youth in the
sample, 11% (8 of 73) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 73 youth in the
sample, 3% (2 of 73) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 26 youth
High = 31 youth
Total Youth in Sample = 73 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 31 youth
Total Youth in Sample = 73 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (66 of 73) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (31 of 73) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 78 100 78%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Marion Youth
Academy.
76 youth completed the program during the review period. Of those youth, 73 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 73 total youth sampled, only
43 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 8
received at least the recommended weeks of service. Youth in the sample completed between 12
and 25 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 0 points for Amount of Service: Contact Hours. Of the 73 total youth
sampled, only 43 included dosage in the EBS Module. Of those youth with correct dosage, 2
received at least the recommended hours of service. Youth in the sample completed between 25
and 51 hours of service, with an average of 29 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Anger Management for Mental Health and Substance
Abuse Clients and The EQUIP Program Combined Service
SPEP Review Date(s): 6/30/2020
Florida Department of Juvenile Justice
Report Date(s): 6/30/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Mental Health and Substance
Abuse Clients and The EQUIP Program Combined Service.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized both the Anger Management for Substance Abuse and Mental Health Clients
and The EQUIP Program curricula during the review period, as a temporary primary service
designation. A review of the documentation for the annual review period of March 2019 through
March 2020 showed there are six staff trained to facilitate The EQUIP Program and eight staff
trained to facilitate Anger Management for Substance Abuse and Mental Health Clients. Six of the
eight staff trained in Anger Management for Substance Abuse and Mental Health Clients
facilitated groups.
The training for The EQUIP Program is a four-hour in-service training provided by the clinical
director. The Anger Management for Substance Abuse and Mental Health Clients training is an
in-service training conducted by the clinical director.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Anger Management for Substance Abuse and Mental Health Clients
manual. The manual includes twelve sessions. Each lesson has a script for the facilitator and
instructions for each activity. The manual requires the lessons be given in order and provides
specific instructions on the implementation of the program. There are no separate student
workbooks. The facilitators make copies of the current lessons to hand out to the youth.
The program utilizes The EQUIP Program: Teaching Youth to Think and Act Responsibly through
a Peer-Helping Approach manual. The manual is broken down into seven sections. The first
section discusses how to implement the EQUIP program. The second section contains the
assessment and individual performance plans. Section three discusses mutual help meetings and
section four discusses the equipment meetings. Section five discusses anger management and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
thinking errors. Section six discusses social skills and section seven discusses social decision
making.
The program has a policy in place to ensure all youth are receiving both interventions.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of The EQUIP Program nor Anger Management for Substance Abuse and Mental
Health Clients was able to be observed due to the COVID-19 pandemic; therefore, this indicator is
rated as non-applicable.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 2019 through March 2020
showed there was facilitator turnover which adversely affected the delivery of Anger Management
for Substance Abuse and Mental Health Clients and The EQUIP Program. Two of the trained
facilitators are no longer employed at the program. One of the trained facilitators left after the
completion of the cohort. The other trained facilitator left prior to the completion of the cohort.
There were no gaps in the primary service due to another trained facilitator taking over for the
facilitator which left the program.
There was documentation of the program completing six cohorts of Anger Management for
Substance Abuse and Mental Health Clients during the review period as evidenced by the sign in
sheets. There is currently one cohort being run during the time of the review. There was
documentation of the program completing seven cohorts of The EQUIP Program during the
review period as evidenced by the sign in sheets. There were two cohorts in progress at the end
of the review period.
The documentation showed each cohort began with ten youth but all ended with less youth than it
began with. The cohort which began on 3/5/2019 had five youth released prior to completing the
primary service and one youth sent to detention. The cohort which began on 4/14/2019 had six
youth discharged prior to completing the primary service. The cohort which began on 5/7/2019
had five youth discharged and one youth sent to detention prior to successfully completing the
cohort. The cohort which began on 6/18/2019 had nine of the ten youth discharged prior to
successfully completing the primary service. The cohort which began on 9/3/2019 had five youth
discharged prior to successfully completing the primary service. The cohort which began on
10/16/20019 had four youth discharged prior to successfully completing the primary service. The
cohort which began on 11/14/2019 had six youth discharged prior to successfully completing the
primary service.
The clinical director stated the program's policy for turnover is to ensure everyone is trained in the
SPEP curriculum. If there is facilitator turnover, then another trained facilitator will take over the
group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation reviewed showed there were thirty-nine internal fidelity checks completed for The
EQUIP Program and twenty-four fidelity checks completed for Anger Management for Substance
Abuse and Mental Health Clients. The clinical director completed each fidelity check on all
facilitators for each month they facilitated the primary service.
The clinical director stated the program's policy for fidelity monitoring is for the clinical director to
complete monthly fidelity monitoring for the mental health services. The director of case
management or other trained facilitator will conduct the fidelity monitoring for the other services.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the clinical director, the program's policy for corrective actions noted on the fidelity
monitoring is for the clinical director to sit down with the therapist to discuss the feedback, provide
written strategies and goals with the facilitator and discuss the facilitator's strengths and
weaknesses. The clinical director will then create two to three goals or strategies for the facilitator.
The clinical director will increase the fidelity monitoring to weekly until the corrective action is
satisfied.
The documentation reviewed during the review period showed no corrective actions were noted
nor taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of faciliatator evaluations for both Anger Management for Substance Abuse and Mental
Health Clients and The EQUIP Program confirmed each have an area which addresses the
facilitator's skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 72 youth in the
sample, 76% (55 of 72) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 40 contact hours for this type of service. Of the 72 youth in the
sample, 53% (38 of 72) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 28 youth
High = 30 youth
Total Youth in Sample = 72 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 30 youth
Total Youth in Sample = 72 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (64 of 72) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (30 of 72) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 83 95 87%
This SPEP report evaluates Anger Management for Mental Health and Substance Abuse Clients
and The EQUIP Program Combined Service, an intervention delivered at Marion Youth Academy.
74 youth completed the program during the review period. Of those youth, 72 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 72 total youth sampled, only
17 included dosage with end dates in the EBS Module, because of an error in the Juvenile Justice
Information System – Evidence-Based Services Module. A manual audit of the remaining youth
was conducted to include dosage calculations. Of the youth in the sample, 55 received at least
the recommended weeks of service. Youth in the sample completed between 20 and 26 weeks of
service, with an average of 23 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 72 total youth
sampled, only 71 included dosage in the EBS Module. Due to an error in the Juvenile Justice
Information System – Evidence-Based Services Module, a manual audit of the youth receiving
The EQUIP Program was conducted to include dosage calculations. Of the youth in the sample,
38 received at least the recommended hours of service. Youth in the sample completed between
20 and 53 hours of service, with an average of 40 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Date(s): 6/29/2020
Florida Department of Juvenile Justice
Report Date(s): 6/29/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 2019 through March 2020
showed there were two staff trained to facilitate Impact of Crime (IOC). IOC training is four-days,
provided by a qualified trainer. The attendance sheets provided showed there was only one staff
member facilitating all IOC groups. The other trained staff member is the director of case
management.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime, Addressing the Harm manual. The manual includes
seven sections broken down into lessons. Each lesson has a script for the facilitator and
instructions for each activity. The manual requires that the lessons be given in order. The manual
provides specific instructions on the implementation of the program. There are no separate
student workbooks. The facilitators make copies of the current lessons to hand out to the youth.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed due to the COVID-19 pandemic;
therefore, this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period showed there was no Impact of Crime
facilitator turnover at the program. There was documentation of the program completing three
cohorts during the review period as evidenced by the sign in sheets. The third cohort began on
2/8/2020 and extended past the annual review period.
The clinical director stated the program's policy for turnover is to ensure everyone is trained in the
SPEP curriculum. If there is facilitator turnover, then another trained facilitator will take over the
group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period shows there were thirteen internal fidelity
checks completed. The fidelity checks were completed each month, without exception.
The clinical director stated the program's policy for fidelity monitoring is for the clinical director to
complete monthly fidelity monitoring of the mental health services. The director of case
management or other trained facilitator will conduct fidelity monitoring for the other services.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the clinical director, the program's policy for corrective actions noted on the fidelity
monitoring is that the clinical director will sit down with the therapist to go over the feedback. The
clinical director will provide written strategies and goals with the facilitator, discuss strengths and
weaknesses, then create two to three goals or strategies for the facilitator. The clinical director will
increase the fidelity monitoring to weekly until the corrective action is satisfied.
The documentation reviewed during the review period showed no corrective actions noted nor
taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program had staff evaluations available for review. All
evaluations reviewed have an area which addresses the facilitator's skill in Impact of Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 68 youth in the
sample, 37% (25 of 68) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 68 youth in the
sample, 38% (26 of 68) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 26 youth
High = 28 youth
Total Youth in Sample = 68 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 28 youth
Total Youth in Sample = 68 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (60 of 68) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 41% (28 of 68) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Marion Youth Academy.
74 youth completed the program during the review period. Of those youth, 68 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 68 total youth sampled, only
28 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 25
received at least the recommended weeks of service. Youth in the sample completed between 14
and 23 weeks of service, with an average of 18 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 68 total youth
sampled, only 28 included dosage in the EBS Module. Of those youth with correct dosage, 26
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
received at least the recommended hours of service. Youth in the sample completed between 21
and 34 hours of service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can maintain their SPEP Quality of Service Delivery score by continuing
the practices in place at the time of this review.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8