summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Thinking for a Change
SPEP Review Date(s): 6/30/2020
Florida Department of Juvenile Justice
Report Date(s): 6/30/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 2019 through March 2020
showed there are five staff trained to facilitate Thinking for a Change (T4C). T4C training is a
thirty-two-hour training completed by a qualified trainer. The documentation reviewed showed
there were three staff conducting cohorts during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Thinking for a Change 4.0 manual. The manual includes twenty-five
lessons. Each lesson has a script for the facilitator and instructions for each activity. The manual
requires the lessons be given in order. The manual provides specific instructions on the
implementation of the program. There are no separate student workbooks. The facilitators make
copies of the current lessons to hand out to the youth.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was unable to be observied due to the COVID-19 pandemic;
therefore, this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 2019 through March 2020
showed there was facilitator turnover; however, the turnover did not adversely affect the delivery
of Thinking for a Change. One trained facilitator is no longer working at the program. The
facilitator completed the cohort prior to leaving the program. The next cohort began with another
trained facilitator.
There was documentation of the program completing two cohorts during the review period; one
cohort was in progress on 3/31/2020, the end of the review period.
The first cohort began 3/15/2019 with twelve (12) youth and ended 6/6/2019 with nine youth. One
youth received new charges and the two other youth were released from the program. The
second cohort began 7/23/2019 with twelve (12) youth and ended 11/18/2019 with eight youth.
The third cohort began on 12/19/2019 with twelve (12) youth. At the time of the annual review, five
youth had been released from the program prior to completing the primary service.
There was a gap of over one month between the cohort which ended on 6/6/2019 and the one
which began on 7/23/2019. There was another gap of one month between the cohort which
ended on 11/18/2019 and the which began on 12/19/2019.
The clinical director stated the program's policy for turnover is to ensure that everyone is trained
in the SPEP curriculum. If there is facilitator turnover, another trained facilitator will take over the
group.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed showed internal fidelity checks were completed each month of the
review period, without exception.
The clinical director stated the program's policy for fidelity monitoring is for the clinical director to
complete monthly fidelity monitoring for the mental health services. The director of case
management or other trained facilitator will conduct the fidelity monitoring for the other services.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation provided for Thinking for a Change showed no corrective actions were noted
nor taken.
According to the clinical director, the program's policy for corrective actions noted on the fidelity
monitoring is that the clinical director will sit down with the therapist to go over the feedback. The
clinical director will provide written strategies and goals with the facilitator, discuss the strengths
and weaknesses, then create two to three goals or strategies for the facilitator. The clinical
director will increase the fidelity monitoring to weekly until the corrective action is satisfied.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed all evaluations have an area which addresses the facilitator's
skill in delivering the primary service, Thinking for a Change.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 70 youth in the
sample, 10% (7 of 70) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 70 youth in the
sample, 10% (7 of 70) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 26 youth
High = 30 youth
Total Youth in Sample = 70 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 43%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 30 youth
Total Youth in Sample = 70 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (62 of 70) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 43% (30 of 70) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 68 100 68%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Marion Youth
Academy.
74 youth completed the program during the review period. Of those youth, 70 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 70 total youth sampled, only
35 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended weeks of service. Youth in the sample completed between 9
and 17 weeks of service, with an average of 13 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 70 total youth
sampled, only 35 included dosage in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended hours of service. Youth in the sample completed between 24
and 51 hours of service, with an average of 37 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Marion Regional Juvenile Detention Center

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Anger Management for Mental Health and Substance
Abuse Clients and The EQUIP Program Combined Service
SPEP Review Period: 4/1/2020-3/31/2021
SPEP Review Conducted: 5/17/2021-5/18/2021
Florida Department of Juvenile Justice
Report Date: 6/30/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Mental Health and Substance
Abuse Clients and The EQUIP Program Combined Service.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized both the Anger Management for Substance Abuse and Mental Health Clients
and The EQUIP Program curricula during the review period, as a temporary primary service
designation. A review of documentation for the annual review period of April 1, 2020 through
March 31, 2021 showed there were five staff trained to facilitate both Anger Management for
Substance Abuse and Mental Health Clients and The EQUIP Program. The trained staff are five
therapists. There were five staff who facilitated or co-facilitated a cohort of Anger Management for
Substance Abuse and Mental Health Clients during the annual review period. Four of the five
trained staff facilitated The EQUIP Program during the annual review period. The documentation
for The EQUIP Program showed there was one staff who was listed as the facilitator on two of the
attendance sheets during the month of July 2020 who did not have proof of training. The staff
member was a mental health intern. The new clinical director stated she did not know who that
person was since that was when the former clinical director was at the program.
The Anger Management for Substance Abuse and Mental Health Clients training is a four-hour in-
service training provided by the clinical director. The EQUIP Program training is a four-hour in-
service training provided by the clinical director. The clinical director conducts the monthly internal
fidelity checks.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program utilizes the Anger Management for Substance Abuse and Mental Health
Clients manual. The manual includes twelve sessions. Each lesson has a script for the facilitator
and instructions for each activity. The manual requires the lessons be given in order and provides
specific instructions on the implementation of the program. The service is a closed group. There
are no separate student workbooks. The facilitators make copies of the current lessons to hand
out to the youth.
The program utilizes the EQUIP Program: Teaching Youth to Think and Act Responsibly through
a Peer-Helping Approach manual. The manual is broken down into seven sections. The first
section discusses how to implement the EQUIP program. The second section contains the
assessment and individual performance plans. Section three discusses mutual help meetings and
section four discusses the equipment meetings. Section five discusses anger management and
thinking errors. Section six discusses social skills and section seven discusses social decision
making. The group is an open group with no more than twelve youth in each cohort.
The program keeps all SPEP documents in two separate binders. One binder is labeled EQUIP.
The other binder is labeled Anger Management. The program provides both Anger Management
for Substance Abuse and Mental Health clients on Tuesdays and Thursdays for one hour each.
The EQUIP Program is ran year-round. The youth are placed in The EQUIP Program until a
cohort of Anger Management for Substance Abuse and Mental Health Clients is opened. Youth
who have not completed Anger Management for Substance Abuse and Mental Health Clients will
be added to the new cohort. The youth will go back into The EQUIP Program group upon
completion of the Anger Management for Substance Abuse and Mental Health Clients service.
The bundled service is to be delivered to the youth over a minimum of twenty-four weeks for forty-
hours.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of The EQUIP Program was observed on 5/18/2021. There were nine youth present for
the group along with the facilitator and one additional staff member. The group began at 3:24pm
and was held in the Cafeteria. The facilitator was knowledgeable of the material and followed the
lesson plan. There were several disruptions during group. The youth would hold side
conversations during group. The facilitator did not speak over the youth. The facilitator waited on
the youth to finish their conversations before restarting group. There was one youth who was
trying to rile up another youth. The youth would call the other youth names and would get up and
then pretend to slap the youth in the head. The other youth would get up and move. The
antagonizing youth would follow the other youth. The youth then took off his shirt and made it
appear that he wanted to fight the youth. The floor staff did not do much to stop the youth from
aggravating the other youth. The facilitator was prepared for the group. The group ended at 4:15p.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Technical assistance specialist spoke with the facilitator about the youth. The facilitator stated she
didn’t remove the youth because that would be what he wanted.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed there was facilitator turnover which adversely affected the delivery of Anger Management
for Substance Abuse and Mental Health Clients and The EQUIP Program. Two of the five trained
facilitators are no longer employed at the program. One of the trained facilitators left after the
completion of an Anger Management for Substance Abuse and Mental Health Clients cohort. The
other trained facilitator who left the program was facilitating The EQUIP Program, which is an
open group. There were no gaps in the primary service due to another trained facilitator taking
over for the facilitators who left the program. There was documentation of the program completing
seven cohorts of Anger Management for Substance Abuse and Mental Health Clients during the
review period as evidenced by the sign in sheets. The dates of the cohorts were: prior to April
2020-4/14/2020, 4/7/2020-6/16/2020, 9/22/2020-12/3/2020, 9/1/2020-11/5/2020, 12/8/2020-
3/2/2021, 11/17/2020-2/16/2021, and 2/23/2021-5/2/2021.
The cohort which began prior to April 2020 began with ten youth and ended with ten youth. The
cohort which began on 4/7/2020 began with nine youth and ended with seven youth. The cohort
which began on 9/22/2020 began with nine youth ended with seven youth. The cohort which
began on 9/1/2020 began with eight youth and ended with five youth. The cohort which began on
12/8/2020 began with nine youth and ended with six youth. The cohort which began on
11/17/2020 began with ten youth and ended with ten youth. The cohort which began on 2/23/2021
began with ten youth and ended with ten youth.
The clinical director stated the program's policy for turnover is to ensure everyone is trained in the
SPEP curriculum. If there is facilitator turnover, then another trained facilitator will take over the
group.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program completed twenty-one monthly internal fidelity monitoring for Anger
Management for Substance Abuse and Mental Health Clients.
The documentation reviewed for the annual review period showed the program completed
nineteen internal fidelity monitorings for The EQUIP Program. The program missed internal fidelity
checks in May 2020 for one therapist, August 2020 for two therapists, September 2020 for two
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
therapists, and November 2020 for two therapists. The clinical director stated the program was
transitioning from the former clinical director who left on 7/10/2020 to the new clinical director who
began on 9/1/2020.
The clinical director stated the program's policy for fidelity monitoring is that the clinical director
completes monthly fidelity monitoring for the mental health services.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for the review period showed there were no corrective actions
noted on the fidelity monitoring. The program's policy for corrective actions noted on the fidelity
monitoring is that the director of case management will sit down with the facilitator to go over the
feedback. The director of case management will provide written or verbal strategies and goals
with the facilitator. The director of case management will increase the fidelity monitoring to
weekly, if needed, until the corrective action is ameliorated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed all evaluations reviewed have an area which addresses the facilitator's skill in the primary
service. The technical assistance specialist reviewed three annual performance evaluations. Each
of the annual performance evaluations contained specific information documenting the facilitator’s
skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 44 youth in the
sample, 30% (13 of 44) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 40 contact hours for this type of service. Of the 44 youth in the
sample, 43% (19 of 44) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 11 youth
High = 22 youth
Total Youth in Sample = 44 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 22 youth
Total Youth in Sample = 44 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (38 of 44) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 50% (22 of 44) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 69 95 73%
This SPEP report evaluates Anger Management for Mental Health and Substance Abuse Clients
and The EQUIP Program Combined Service, an intervention delivered at Marion Youth Academy.
44 youth completed the program during the review period. Of those youth, all 44 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 44 total youth sampled, only
30 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 9
and 27 weeks of service, with an average of 20 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 4 points for Amount of Service: Contact Hours. Of the 44 total youth
sampled, only 32 included dosage in the EBS Module. Of those youth with correct dosage, 19
received at least the recommended hours of service. Youth in the sample completed between 21
and 52 hours of service, with an average of 39 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring all
facilitators receive formal training prior to delivering services to youth, facilitator turnover does not
impact groups, and fidelity monitoring is completed every month for each facilitator.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Marion Youth Academy

program_status_note: Closed


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Period: 4/1/2020-3/31/2021
SPEP Review Conducted: 5/17/2021-5/18/2021
Florida Department of Juvenile Justice
Report Date: 7/1/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed there were eight staff trained to facilitate Impact of Crime (IOC). The trained staff are the
director of case management, clinical director, three therapists, one case manager, transition
specialist, and one lead shift supervisor. There were four staff who facilitated or co-facilitated a
cohort of the primary service during the annual review period. The training is an in-person training
which is twenty-eight hours long conducted over three days by a qualified trainer. The primary
service is a closed group. The director of case management conducted the internal fidelity checks
for the annual review period. All trained staff members can conduct the internal fidelity checks.
The clinical director stated the therapists have been unable to facilitate the primary service due to
scheduling conflicts with the required mental health groups. Impact of Crime cannot be substituted
for a mental health group so the therapists cannot facilitate IOC at this time.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program utilizes the Impact of Crime: Addressing the Harm manual. The manual
opens with an introduction to the Impact of Crime Curriculum. Then the manual goes over all
seven chapters of the manual and provides a script for the facilitator to follow. The manual also
has an appendix and a student guide with all the student exercises. The program does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth. The group is a closed group with no more than twelve youth in
each cohort. The curriculum is to be delivered to the youth over a minimum of sixteen weeks for
thirty-two sessions. The program keeps all SPEP documents in a binder labeled Impact of Crime.
The program provides the service on Tuesdays and Thursdays for one hour each. All youth
receive Impact of Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 5/18/2021. There were seven youth present for the
group along with the facilitator. The group began at 11:50a. The group was held in the cafeteria.
The facilitator was knowledgeable of the material and followed the lesson plan. Most of the youth
participated during the group. Occasionally, a youth would try to make jokes during the group.
There were minor disruptions from the youth during group. The facilitator was not prepared for the
group but was knowledgeable of the material. He had to go make copies of the group activity
during the lesson. The group ended at 12:50p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed there was facilitator turnover for Impact of Crime. There were two cohorts completed and
one ongoing at the time of this review. The dates of the cohorts are: 2/8/2020-5/29/2020,
10/5/2020-4/27/2021, and 3/8/2021-current. There was a gap over four months between the
cohort which ended on 5/29/2020 and the cohort which began on 10/5/2020. The clinical director
and director of case management stated the gap was because the program had to negotiate with
the school to get time for the group. The program was also dealing with low census during that
time. The director of case management stated the program's policy for turnover is to ensure all
case management staff is trained in the SPEP curriculum. If there is facilitator turnover, then
another trained facilitator will take over the group.
The cohort which began on 2/8/2020 began with twelve youth and ended with eight youth. The
facilitator ran back to back groups on 2/16/2020 and 2/17/2020 and then again on 5/17/2020 and
5/18/2020.
The cohort which began on 10/5/2020 began with twelve youth and ended with two youth. The
director of case management and clinical director stated the facilitator had to repeat multiple
lessons due to the youth’s processing ability. The youth in the cohort had a difficult time
understanding the topics. The facilitator also had difficulty finding a community service project for
the youth to complete. The facilitator ran back to back lessons on 12/9/2020 and 12/10/2020.
There was a note on the attendance sheets for 1/5/2021, 1/7/2021, and 1/11/2021 which stated
the group did not happen due to COVID-19.
The cohort which began on 3/8/2021 began with twelve youth and is ongoing as of this report.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program completed eleven fidelity monitorings, accounting for each facilitator during
each month groups were held. The director of case management conducted the monthly internal
fidelity checks during the annual review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation for the review period showed there were no corrective actions noted
on the fidelity monitoring of Impact of Crime. The program's policy for corrective actions noted on
the fidelity monitoring is that the director of case management will sit down with the facilitator to
go over the feedback. The director of case management will provide written or verbal strategies
and goals with the facilitator. The director of case management will increase the fidelity monitoring
to weekly, if needed, until the corrective action is ameliorated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed all evaluations reviewed have an area which addresses the facilitator's skill in the primary
service. The technical assistance specialist reviewed four annual performance evaluations. Each
of the annual performance evaluations contained specific information documenting the facilitator’s
skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 44 youth in the
sample, 27% (12 of 44) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 44 youth in the
sample, 34% (15 of 44) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 11 youth
High = 23 youth
Total Youth in Sample = 44 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 52%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 23 youth
Total Youth in Sample = 44 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (39 of 44) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 52% (23 of 44) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Marion Youth Academy.
44 youth completed the program during the review period. Of those youth, all 44 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 44 total youth sampled, only
16 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 12
received at least the recommended weeks of service. Youth in the sample completed between 9
and 24 weeks of service, with an average of 16 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 44 total youth
sampled, only 16 included dosage in the EBS Module. Of those youth with correct dosage, 15
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended hours of service. Youth in the sample completed between 18
and 43 hours of service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Thinking for a Change
SPEP Review Period: 4/1/2020-3/31/2021
SPEP Review Conducted: 5/17/2021-5/18/2021
Florida Department of Juvenile Justice
Report Date: 7/1/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed there were three staff trained to facilitate Thinking for a Change (T4C). The trained staff
are the director of case management, one case manager, and the transition specialist. There
were two staff who facilitated or co-facilitated a cohort of the primary service during the annual
review period. The primary service training is a four-day training conducted by a qualified trainer.
There was one staff who facilitated the primary service during the annual review period. The
primary service is a closed group. The director of case management conducts the monthly
internal fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program utilized the Thinking for a Change 4.0 manual. The manual includes 25
lessons. Each lesson has a script for the facilitator and instructions for each activity. The manual
requires the lessons be given in order and provides specific instructions for the implementation of
the program. There are no separate student workbooks. The facilitators make copies of the
current lessons to hand out to the youth. The group is a closed group with no more than twelve
youth in each cohort. The curriculum is to be delivered to the youth over a minimum of fifteen
weeks for forty-five hours. The program keeps all SPEP documents in a binder labeled Thinking
for a Change. The program provides the service on Mondays, Wednesdays, and Fridays for one
hour each. All youth receive Thinking for a Change.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed on 5/17/2021. There were eight youth present
for the group along with the facilitator and the clinical director. The group began at 11:09a. The
group was held on the Phase one recreation area. The facilitator was knowledgeable of the
material and followed the lesson plan but did not demonstrate the role play process properly. The
facilitator did not demonstrate pointing to the head to identify a thinking step. The youth were not
assigned a skill step to look for during the role play. There were disruptions during group. Lunch
was passed out to the youth during the group. There were several staff walking through the
recreation area during group which caused disruptions. The facilitator was prepared for the group.
The group ended at 12:13p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed there was no facilitator turnover. There were two cohorts completed and two currently
running during the annual review period. The dates of the cohorts are: 4/2/2020-6/4/2020,
10/5/2020-3/22/2021, 2/23/2021-current, and 3/17/2021-current. There was a gap of four months
between the cohort which ended on 6/4/2020 and the cohort which began on 10/5/2020. The
clinical director and director of case management stated the gap was because the program had to
negotiate with the school to get time for the group. The program was also dealing with low census
during that time. The director of case management stated the program's policy for turnover is to
ensure all case management staff is trained in the SPEP curriculum. If there is facilitator turnover,
then another trained facilitator will take over the group.
The 4/2/2020 cohort began with twelve youth and ended with one youth.
The 10/5/2020 cohort began with twelve youth and ended with one youth. The director of case
management and clinical director stated the facilitator had to repeat multiple lessons due to the
youth’s processing ability. The youth in the cohort had a difficult time understanding the topics.
The 2/23/2021 cohort began with nine youth. One youth was released on 4/24/2021.
The 3/17/2021 cohort began with ten youth. Two youth were added to the group during the
second lesson.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed the program completed ten fidelity monitorings of Thinking for a Change, accounting for
each facilitator each month groups were held. The director of case management conducted the
monthly internal fidelity checks during the annual review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for the review period showed there were no corrective actions
noted on the fidelity monitoring of Thinking for a Change. The program's policy for corrective
actions noted on the fidelity monitoring is that the director of case management will sit down with
the facilitator to go over the feedback. The director of case management will provide written or
verbal strategies and goals with the facilitator. The director of case management will increase the
fidelity monitoring to weekly, if needed, until the corrective action is ameliorated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
showed all evaluations reviewed have an area which addresses the facilitator's skill in delivering
Thinking for a Change. The technical assistance specialist reviewed three annual performance
evaluations. Each of the annual performance evaluations contained specific information
documenting the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 41 youth in the
sample, 27% (11 of 41) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 45 contact hours for this type of service. Of the 41 youth in the
sample, 27% (11 of 41) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 21 youth
Total Youth in Sample = 41 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 51%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 21 youth
Total Youth in Sample = 41 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (36 of 41) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 51% (21 of 41) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 82 100 82%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Marion Youth
Academy.
44 youth completed the program during the review period. Of those youth, 41 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 41 total youth sampled, only
14 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 11
received at least the recommended weeks of service. Youth in the sample completed between 7
and 25 weeks of service, with an average of 19 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Contact Hours. Of the 41 total youth
sampled, only 14 included dosage in the EBS Module. Of those youth with correct dosage, 11
received at least the recommended hours of service. Youth in the sample completed between 17
and 68 hours of service, with an average of 50 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Marion Youth Academy
Sequel Youth and Family Services
(Contract Provider)
10420 NW Gainesville Road
Ocala, Florida 34482
Primary Service: Impact of Crime
SPEP Review Period: April 1, 2021-March 31, 2022
SPEP Review Conducted: April 6-7, 2021
Florida Department of Juvenile Justice
Report Date: 7/27/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 1, 2021 through March 31, 2022
showed there were eight staff trained to facilitate Impact of Crime (IOC)- the director of case
management, clinical director, two therapists, two case managers, one recreation therapist, and
one transition specialist. Two case managers and the transition specialist facilitated or co-
facilitated IOC during the annual review period. The training is twenty-eight (28) hours long
conducted over three days by a qualified trainer. The director of case management conducted
internal fidelity checks. All trained staff can conduct internal fidelity checks. The clinical director
stated the therapists have been unable to facilitate the primary service due to scheduling conflicts
with the required mental health groups. Impact of Crime cannot be substituted for a mental health
group so the therapists cannot facilitate IOC at this time.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all seven
chapters of the service and provides a script for the facilitator to follow. The manual also has an
appendix and a student guide with all the student exercises. The program makes copies of the
student exercises and passes them out to the youth during each lesson.
The program conducts IOC on Tuesdays and Thursdays. The program keeps all SPEP
documentation (attendance sheets, fidelity checks, and training certificates) in a binder labeled
"IOC SPEP documentation" in the case management office. All youth admitted to the program are
required to participate in the primary service. According to the clinical director, the program
decides which youth receives the intervention by first determining if the youth has previously
participated in the completion of the required dosage of services. If the youth has not, the youth is
automatically included in the next incoming cohort. If the youth has completed the required
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
dosage, but continues to struggle with personal accountability for their actions, they can be
readmitted into the next incoming cohort. The primary service is a closed group with no more than
twelve (12) youth assigned to one cohort.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 4/7/2022. Present were twelve (12) youth, the
facilitator, and one floor staff. The group began at 11:47am and was held in the cafeteria. The
facilitator was knowledgeable of the material and followed the lesson plan. Most of the youth
participated during the group. The facilitator was prepared for the group. The facilitator ensured
the youth understood the concepts before moving on to another topic. The group ended at
12:32p.
The technical assistance specialist spoke with the clinical director about the group not going for a
full hour. The floor staff stated the youth needed to get back to the classrooms even though the
group did not begin on time. The technical assistance specialist stated all groups need to be ran
for at least one hour. The program cannot put one hour into the Evidence-Based Services module
(EBS) if the group was not run for a full hour. The clinical director stated she will speak with the
facilitator and ensure he runs group for the full hour.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period of April 1, 2021 through March 31, 2022
showed there was turnover of Impact of Crime (IOC) facilitators. Two case managers left the
program. The turnover did not negatively affect delivery of IOC. Cohorts were finished prior to
leaving the program. There were no gaps in service.
There were five cohorts completed and one currently running during the annual review period.
The dates of the cohorts are: 10/5/2020-4/27/2021, 3/8/2021-7/22/2021, 5/18/2021-9/7/2021,
11/17/2021-3/31/2022, 9/14/2021-12/31/2021, and 3/23/2022-current.
The 10/5/2020 cohort began with twelve (12) youth and ended with two youth. The director of
case management and clinical director stated the facilitator had to repeat multiple lessons due to
the youths' processing ability. The youth in the cohort had a difficult time understanding the topics.
The facilitator also had difficulty finding a community service project for the youth to complete.
The 3/8/2021 cohort began with twelve (12) youth and ended with seven youth. Five of the youth
were missing signatures on multiple attendance sheets. The attendance sheets did not
specifically state if those five youth were released or transferred until the last attendance sheet.
Those youth were released prior to completing the primary service.
The 5/18/2021 cohort began with ten (10) youth and ended with two youth. Seven youth were
released prior to completing the primary service; one youth was arrested.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The 11/17/2021 cohort began with ten (10) youth and ended with seven youth. Three youth were
arrested (two on 1/1/2022 and one on 2/25/2022). There were two missing attendance sheets for
3/15/2022 and 3/17/2022. The clinical director pulled up the video to show the groups were run on
those two days. The technical assistance specialist noted the group on 3/15/2022 was only run for
forty-five (45) minutes and the group on 3/17/2022 was only run for thirty-eight (38) minutes.
The 9/14/2021 cohort began with nine youth and ended with six youth. One youth was released
on 9/16/2021, one youth was released on 9/24/2021, and one youth was released on 10/7/2021.
The 3/23/2022 cohort began with twelve (12) youth and currently has twelve (12) youth.
The clinical director stated the program implements a co-facilitator guideline in which there are
two facilitators facilitating each group, so in the event one of the facilitators is out for any reason,
the group can continue. Therapists are also trained in the primary service in the event that there
are no case managers available to facilitate a group. The technical assistance specialist informed
the clinical director the groups should be run for a full hour. The attendance sheets show the
groups are being ran for one hour, but the observation on 4/7/2022 and video of a group held on
3/15/2022 and 3/17/2022 show the program is not running the primary service for a full hour.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 1, 2021 through March 31, 2022
showed the program completed eighteen (18) fidelity monitorings for facilitators of Impact of
Crime. The program was missing an internal fidelity monitoring for a case manager for the month
of May 2021. The technical assistance specialist informed the clinical director the program needs
to complete an internal fidelity check even if the technical assistance specialist completed a
fidelity monitoring for the case manager in May 2021. The director of case management
conducted the monthly internal fidelity checks during the annual review period. The clinical
director stated the program conducts internal fidelity monitoring on a monthly basis. Facilitators
trained must be monitored by another trained and experience facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the documentation for the annual review period showed there were no corrective
actions noted on the fidelity monitoring of Impact of Crime. The program's policy for corrective
actions noted on the fidelity monitoring is the director of case management will sit down with the
facilitator to go over the feedback. The director of case management will provide written or verbal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
strategies and goals with the facilitator. The director of case management will increase the fidelity
monitoring to weekly, if needed, until the corrective action is ameliorated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 1, 2021 through March 31, 2022
showed all evaluations reviewed have an area which addresses the facilitator's skill in delivering
Impact of Crime (IOC). Three annual performance evaluations were provided for review. One of
the three annual performance evaluations contained specific information documenting the
facilitator’s skill delivering IOC. The other two evaluations did not contain specific information
regarding the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service..
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 46 youth in the
sample, 54% (25 of 46) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 46 youth in the
sample, 57% (26 of 46) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 18 youth
High = 15 youth
Total Youth in Sample = 46 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 15 youth
Total Youth in Sample = 46 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (39 of 46) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 33% (15 of 46) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 68 85 80%
This SPEP report evaluates Impact of Crime, an intervention delivered at Marion Youth Academy.
50 youth completed the program during the review period. Of those youth, 46 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 46 total youth sampled, only
30 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 25
received at least the recommended weeks of service. Youth in the sample completed between 1
and 30 weeks of service, with an average of 18 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 46 total youth
sampled, only 30 included dosage in the EBS Module. Of those youth with correct dosage, 26
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
received at least the recommended hours of service. Youth in the sample completed between 1
and 62 hours of service, with an average of 35 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Marion Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Marion Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Marion IOC.pdf