summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Impact of Crime
SPEP Review Period: 4/15/2020-4/14/2021
SPEP Review Conducted: 5/5/2021-5/6/2021
Florida Department of Juvenile Justice
Report Date: 7/29/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed the program utilizes Impact of Crime (IOC) as a primary service. The documentation
showed there were five staff trained in the primary service. The staff trained are two case
managers, one youth care worker, the transition specialist, and another staff member. Training for
IOC is three days, conducted by a qualified trainer. There was one staff who facilitated the
primary service during the annual review period. Each trained staff member conducts the monthly
internal fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Documentation reviewed for the annual review period of April 15, 2020 through April 14, 2021
confirms the program utilizes the Impact of Crime: Addressing the Harm manual. The manual
opens with an introduction to the Impact of Crime curriculum. Then the manual goes over all
seven chapters of the group and provides a script for the facilitator to follow. The manual also has
an appendix and a student guide with all the student exercises. The program does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth.
The group is a closed group with no more than twelve youth in each cohort. The curriculum is to
be delivered to the youth over a minimum of sixteen weeks for thirty-two sessions. The program
keeps all SPEP documents in a binder labeled Impact of Crime. The program provides the service
on Tuesdays and Thursdays for one hour each. All youth receive Impact of Crime. The clinical
director also stated the program decides which youths receive the intervention by first determining
if the youth has previously participated in the completion of the required dosage of services. If the
youth has not, the youth is automatically included in the next incoming cohort. If the youth has
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
completed the required dosage, but continues to struggle with personal accountability for their
actions, they can be re-admitted into to next incoming cohort.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 5/6/2021. There were six youth present for the
group along with one staff and the facilitator. The group began at 2:33p. The group was held in
the program's multi-purpose room. The youth were making jokes while answering the facilitator’s
questions. The facilitator was knowledgeable of the material and followed the lesson plan. The
group ended at 3:30p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period showed the program did not have
facilitator turnover for Impact of Crime. One facilitator facilitated all four cohorts during the annual
review period. There was one gap in service just under one month between the cohort which
ended on 1/13/2021 and the cohort which began on 2/9/2021. The clinical director stated this gap
in service was due to low census. The program only had one new youth in the program at the time
after the cohort ended on 1/13/2021. The program received five admissions during the three
weeks after 1/13/2021. The documentation reviewed shows there were three cohorts completed
and one cohort currently running. The dates of the cohorts are 2/13/2020-5/27/2020, 5/18/2020-
9/01/2020, 8/31/2020-1/13/2021, and 2/9/2021-present.
The 2/13/2020 cohort began with eleven youth and ended with ten youth. The documentation
shows there were three sessions run back-to-back. Those sessions were on 3/3/2020, 3/4/2020,
and 3/5/2020. The attendance sheet dated 5/6/2020 shows the youth successfully completed the
primary service, but there were four additional attendance sheets dated after 5/6/2020. The last of
the additional attendance sheets has a date of 5/27/2020 which also shows the youth successfully
completed the primary service.
The 5/18/2020 cohort began with fourteen youth and ended with thirteen youth. The last
attendance sheet which was dated on 9/1/2020 showed the cohort was on chapter seven
objective three. The attendance sheets dated on 8/25/2020 and 8/27/2020 show the cohort was
on chapter seven objective five. The documentation did not show the facilitator completing
chapter seven objective four.
The 8/31/2020 cohort began with four youth and ended with two youth. The documentation shows
the primary service was run on back-to-back days three different times during the cohort. The
dates the primary service was run back-to-back were 9/22/2020 and 9/23/2020, 12/21/2020 and
12/22/2020, and 1/12/2021 and 1/13/2021. The attendance sheets dated 11/24/2020, 11/27/2020,
and 11/29/2020 had a note which stated the group was conducted for fifteen minutes due to the
youth being quarantined for COVID-19. The Clinical Director indicated these short sessions were
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
conducted to introduce the youth to the material in advance of a full hour-long session to cover
the entire lesson. The program did not want the youth to get behind in the curriculum.
The 2/9/2021 cohort began with six youth and is currently ongoing at the time of the annual
review.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed there were thirteen internal fidelity checks completed for Impact of Crime, accounting for
each month. One staff facilitated groups during the annual review period and was monitored each
month. Three of the trained facilitators conducted the internal fidelity monitoring. The clinical
director stated the program conducts internal fidelity monitoring on a monthly basis. Facilitators
trained must be monitored monthly by another trained and experienced facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Documentation reviewed for the annual review period showed there were no identified corrective
actions noted during the monthly internal fidelity checks of Impact of Crime. The clinical director
stated when corrective action is needed, the process will be for the facilitator to meet with the
clinical director to review findings from the completed fidelity monitoring to address concerns and
provide support. The facilitator will be re-assessed at the next month’s fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed shows each evaluation has an area called facilitator skill delivering
the evidence-based programming. Four evaluations were reviewed. The transition specialist’s
evaluation did not identify the specific SPEP service the therapist was facilitating. The evaluation
mentioned "all of your SPEP groups are on target to meet the specific dosage", but did not
contain any other specific information. A case manager’s evaluation stated to remember the
SPEP dosage for the two SPEP groups they facilitate. Another case manager and floor staff’s
evaluations were reviewed. The case manager and floor staff were not facilitating the primary
service at the time of the annual review.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 31 youth in the
sample, 52% (16 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 31 youth in the
sample, 90% (28 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 84%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 11 youth
High = 11 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 35%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 11 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 84% (26 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 35% (11 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 72 85 85%
This SPEP report evaluates Impact of Crime, an intervention delivered at Duval Academy.
33 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 31 total youth sampled, only
27 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 1
and 20 weeks of service, with an average of 15 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 28 included dosage in the EBS Module. Of those youth with correct dosage, 28
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended hours of service. Youth in the sample completed between 27
and 55 hours of service, with an average of 41 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Duval Academy

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Living in Balance
SPEP Review Period: 4/15/2020-4/14/2021
SPEP Review Conducted: 5/5/2021-5/6/2021
Florida Department of Juvenile Justice
Report Date: 7/29/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed the program utilizes Living in Balance as a primary service. The documentation shows
there were three staff trained in the primary service- two therapists and the former clinical director.
The training is a four-hour in-service training delivered by the clinical director, assistant clinical
director, or regional clinical director. The program has a written protocol for the in-service training
which documents the schedule for the training. There were two staff who facilitated the primary
service during the annual review period. Each trained staff member conducts the monthly internal
fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of April 15, 2020 through April 14, 2021
confirmed the program conducts Living in Balance (LIB) as an open group and has consistently
held group during the review period. The curriculum is offered to youth who are diagnosed with a
substance abuse disorder and/or substance abuse is one of the top three criminogenic needs on
the youth's residential assessment for youth. The clinical director stated the program also makes
a decision about which youth receives the intervention by first determining if the youth has
previously participated in the completion of the required dosage of services. If the youth has not,
the youth is automatically included in the next incoming cohort. If the youth has completed the
required dosage, but continues to struggle with personal accountability for their actions, they can
be re-admitted into to next incoming cohort. Each manual also includes CDs for facilitator
resources and session handouts. The manuals include facilitation guidelines, specific instructions
for each lesson, and utilizes role plays, video, and discussion. There are a total of forty-seven
lessons for LIB. The sessions are to be sixty to seventy-five minutes in duration. The program
keeps all SPEP documents in a binder labeled Living in Balance. The program provides the
service on Tuesdays and Thursdays for one hour each.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Living in Balance was observed on 5/6/2021. There were nine youth present for the
group along with two staff and the facilitator. The group began at 11:45a. The group was held in
the program's multi-purpose room. The facilitator encouraged all youth to participate in group. The
youth did hold side conversations, but the facilitator was able to get those youth back on track.
Eight of the nine youth were engaged during group. One youth did not want to actively participate
in the group but was not disruptive. The facilitator was knowledgeable of the material and followed
the lesson plan. The youth who participated in group provided good dialogue and answered all the
facilitator's questions. The group ended at 12:45p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 15, 2020 through April 14, 2021
showed the program had facilitator turnover; however, this turnover did not affect the delivery of
Living in Balance. The former clinical director left the program but was not facilitating any groups
during the annual review period. Both staff members who were running the primary service are
still running groups. The clinical director stated the program plans to implement a co-facilitator
guideline in which there are two facilitators facilitating each group, so in the event one of the
facilitators is out for any reason, the group can continue.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed there were twenty-seven internal fidelity checks completed for Living in Balance. There
were two staff who facilitated groups during the annual review period. Both facilitators were
monitored each month. The two trained facilitators conducted the internal fidelity monitoring on
each other. The clinical director stated the program conducts internal fidelity monitoring on a
monthly basis. Facilitators trained must be monitored monthly by another trained and
experienced facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no identified
corrective actions noted during the monthly internal fidelity checks of Living in Balance. When
corrective action is needed, the process will be for the facilitator to meet with the clinical director
to review findings from the completed fidelity monitoring to address concerns and provide support.
The facilitator will be re-assessed at the next month’s fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed shows each evaluation has an area called facilitator skill delivering
the evidence-based programming. None of the evaluations reviewed had specific information
based on the facilitator's delivery of the evidence-based programming. One of the therapist’s
evaluation did not identify the specific SPEP service the therapist was facilitating. The other
therapist’s evaluation mentioned the specific primary service and that the youth were all on track
to have the correct amount of dosage, but did not contain any other specific information.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 32 youth in the
sample, 94% (30 of 32) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 32 youth in the
sample, 97% (31 of 32) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 84%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 11 youth
High = 12 youth
Total Youth in Sample = 32 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 32 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 84% (27 of 32) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 38% (12 of 32) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 86 95 91%
This SPEP report evaluates Living in Balance, an intervention delivered at Duval Academy.
33 youth completed the program during the review period. Of those youth, 32 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 32 total youth sampled, all
32 included dosage with end dates in the EBS Module. Of those youth, 30 received at least the
recommended weeks of service. Youth in the sample completed between 13 and 54 weeks of
service, with an average of 31 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 32 total youth
sampled, all 32 included dosage in the EBS Module. Of those youth, 31 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 22 and 100 hours of
service, with an average of 64 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Duval Academy - Sequel

program_status_note: Closed

service_type: Living in Balance


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Thinking for a Change
SPEP Review Period: 4/15/2020-4/14/2021
SPEP Review Conducted: 5/5/2021-5/6/2021
Florida Department of Juvenile Justice
Report Date: 7/29/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed the program utilizes Thinking for a Change (T4C) as a primary service. The
documentation reviewed showed there were three staff trained in the primary service- two case
managers and the transition specialist. T4C training is a four-day training conducted by a qualified
trainer. There was one staff who facilitated the primary service during the annual review period.
Another trained staff member conducts the monthly internal fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Documentation reviewed for the annual review period of April 15, 2020 through April 14, 2021
confirmed the program utilized the Thinking for a Change 4.0 manual. The manual includes
twenty-five lessons. Each lesson has a script for the facilitator and instructions for each activity.
The manual requires that the lessons be given in order and provides specific instructions for the
implementation of the program. There are no separate student workbooks. The facilitators make
copies of the current lessons to hand out to the youth.
The group is a closed group with no more than twelve youth in each cohort. The curriculum is to
be delivered to the youth over a minimum of fifteen weeks for forty-five hours. The program keeps
all SPEP documents in a binder labeled Thinking for a Change. The program provides the service
on Wednesdays and Fridays for one-and-one-half hour each. All youth receive Thinking for a
Change. The clinical director also stated the program decides which youth receives the
intervention by first determining if the youth has previously participated in the completion of the
required dosage of services. If the youth has not, the youth is automatically included in the next
incoming cohort. If the youth has completed the required dosage, but continues to struggle with
personal accountability for their actions, they can be re-admitted into to next incoming cohort.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed on 5/5/2021. There were seven youth present
for the group along with the facilitator. The group began at 2:32p. The group was held in the
program's multi-purpose room. The youth were making jokes while answering the facilitator’s
questions. The youth were also trying to interrupt the facilitator during the group. There were one
or two youth who kept asking the facilitator how much longer they had for group. The youth did
not appear to grasp the concepts presented during the session. The facilitator asked questions
about prior sessions and the youth couldn’t answer those questions. The facilitator attempted to
keep the youth on track. The facilitator was knowledgeable of the material and followed the lesson
plan. The group ended at 4:05p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation for the annual review period showed the program did not have
facilitator turnover for Thinking for a Change. There was one facilitator who facilitated all five
cohorts during the annual review period. There was one gap in service just under two months
between the cohort which ended on 12/16/2020 and the cohort which began on 2/12/2021. The
clinical director stated this gap in service was due to low census. The program only had one new
youth in the program at the time after the cohort ended on 12/16/2020, but received five additional
admissions during the following weeks, up until 2/11/2021.
The documentation reviewed shows there were four cohorts completed and one cohort currently
running. The dates of the cohorts are 2/19/2020-5/15/2020, 5/29/2020-8/26/2020, 5/29/2020-
8/31/2020, 9/2/2020-12/16/2020, and 2/12/2021-present
The 2/19/2020 cohort began with nine youth and ended with ten youth. One youth was added
during the second session.
The 5/29/2020 cohort began with three youth and ended with two youth. The second 5/29/2020
cohort began with twelve youth and ended with eleven youth.
The 9/2/2020 cohort began with six youth and ended with five youth.
The 2/12/2021 cohort began with seven youth and was ongoing at the time of the annual review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2020 through April 14, 2021
showed there were fifteen internal fidelity checks completed for Thinking for a Change. One staff
facilitated groups during the annual review period and was monitored each month. One of the
trained facilitators who was not facilitating the primary service conducted the internal fidelity
monitoring. The clinical director stated the program conducts internal fidelity monitoring on a
monthly basis. Facilitators trained must be monitored monthly by another trained and
experienced facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Documentation reviewed for the annual review period showed there were no identified corrective
actions noted during the monthly internal fidelity checks of Thinking for a Change. When
corrective action is needed, the process will be for the facilitator to meet with the clinical director
to review findings from the completed fidelity monitoring to address concerns and provide support.
The facilitator will be re-assessed at the next month’s fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed shows each evaluation has an area called facilitator skill delivering
the evidence-based programming. Three evaluations were reviewed. The transition specialist’s
evaluation did not identify the specific SPEP service the therapist was facilitating. The evaluation
mentioned "all of your SPEP groups are on target to meet the specific dosage", but did not
contain any other specific information. A case manager’s evaluation stated to remember the
SPEP dosage for the two SPEP groups they facilitate.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 15 weeks for this type of service. Of the 31 youth in the
sample, 52% (16 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 31 youth in the
sample, 52% (16 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 11 youth
High = 12 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (27 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 39% (12 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 86 100 86%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Duval Academy.
33 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 31 total youth sampled, all
31 included dosage with end dates in the EBS Module. Of those youth, 16 received at least the
recommended weeks of service. Youth in the sample completed between 12 and 16 weeks of
service, with an average of 14 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, all 31 included dosage in the EBS Module. Of those youth, 16 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 30 and 49 hours of
service, with an average of 43 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Impact of Crime
SPEP Review Period: April 15, 2021-April 14, 2022
SPEP Review Conducted: May 25, 2022
Florida Department of Juvenile Justice
Report Date: 8/3/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed the program utilizes Impact of Crime (IOC) as a primary service. The documentation
confirmed there were five staff trained in the primary service: four case managers and the
transition specialist. The training is twenty-eight (28) hours conducted in-person training over
three days by a qualified trainer. All trained staff may conduct the monthly internal fidelity checks.
Two staff facilitated IOC during the annual review period. Two trained staff members (case
manager and transition specialist) conducted the monthly internal fidelity checks.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all seven
chapters of the service and provides a script for the facilitator to follow. The manual also has an
appendix and a student guide section with all the student exercises. The program does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth.
IOC is a closed group with no more than twelve (12) youth in each cohort. The curriculum is to be
delivered to the youth over a minimum of sixteen (16) weeks for thirty-two (32) sessions. The
program keeps all SPEP documents in a binder labeled "Impact of Crime" in the clinical director’s
office. The program provides the service on Tuesdays and Thursdays for one hour each. All youth
receive Impact of Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 5/19/2022. There were twelve (12) youth, one staff
and the facilitator present for group. The group began at 2:30pm and was held in the multi-
purpose room. The facilitator was unable to follow the lesson. The youth were being disrespectful
during the check in and throughout the session. The facilitator asked the youth if they could state
what Stinking Thinking was and the youth stated it was looking at things negatively. The facilitator
was trying to review chapter two before going into the lesson in chapter four. Technical assistance
specialist asked the facilitator to go back over chapter two and review the topics. During the
group, a youth was continuing to be disrespectful, so the facilitator had the youth stand outside of
the room. The facilitator did not check to see if there was a staff member outside of the room to
watch the youth. The group ended at 3:36p.
The technical assistance specialist spoke with the clinical director about the youth being
disrespectful, the facilitator sending a youth outside without checking if there was a staff member
available to watch the youth, and the facilitator needing to review chapters one through three prior
to continuing with chapter four. The technical assistance specialist asked the clinical director if
she would observe another session to determine if the facilitator needs additional effective group
facilitation training. The clinical director stated she would observe the next session and send an
email with her observations.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 15, 2021 showed the program
experienced facilitator turnover; however, the turnover did not adversely affect the delivery of
Impact of Crime. A case manager left the program in June 2021 and another case manager left
the program at the end of October 2021. Neither case managers were facilitating a cohort at the
time they left.
The documentation reviewed shows there were three cohorts completed and two cohorts ongoing
at the time of this review. Two trained staff members (one case manager and transition specialist)
facilitated all five cohorts. The dates of the cohorts are 2/9/2021-6/15/2021, 6/29/2021-
10/21/2021, 10/26/2021-2/22/2022, 3/8/2022-current, and 3/9/2022-current.
The 2/9/2021 cohort began with six youth and ended with five youth. One youth was discharged
prior to completing the primary service.
The 6/29/2021 cohort began with four youth and ended with three youth. One youth was added to
the cohort on 7/6/2021. One youth was discharged on 10/12/2021 prior to completing the primary
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
service. Another youth was dropped from the cohort with no known explanation. The attendance
sheets dated 8/26/2021 shows the facilitator was on chapter five objective one. The next
attendance sheet dated on 8/31/2021 shows the facilitator went back to chapter two objective two.
On 9/2/2021, the facilitator jumped to chapter four objective one. On 9/7/2021, the facilitator went
back to chapter five objective two. The technical assistance specialist asked the transition
specialist about the reason for jumping around. The transition specialist stated he sometimes
forgot which lessons he has covered. The technical assistance specialist stated the transition
specialist could write down the lesson on the attendance sheet so there would be no confusion
moving forward. The transition specialist stated he would do that and also make copies of the
attendance sheets in the future.
The 10/26/2021 cohort began with twelve (12) youth and ended with ten (10) youth. The
attendance sheet dated 11/11/2021 shows the facilitator was on chapter two objective two. The
attendance sheet dated 11/16/2021 shows the facilitator went back to chapter one objective two.
The case manager stated he forgot to finish up chapter one before he moved on to the next
chapter, so he went back to ensure he covered those topics. One youth was arrested on
11/2/2021. One youth was on the hub on 1/25/2022 and then home on 1/27/2022.
The 3/8/2022 cohort began with six youth and currently has five youth. One youth was discharged
on 4/16/2022. The 3/9/2022 cohort began with twelve (12) youth and currently has twelve (12)
youth.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed there were thirteen (13) internal fidelity checks completed for facilitators of Impact of
Crime. The program was missing a fidelity check for the months of October 2021, January 2022,
and February 2022 for the case manager. Two of the trained facilitators conducted the internal
fidelity monitoring. The clinical director stated the program conducts internal fidelity monitoring on
a monthly basis. Facilitators trained must be monitored monthly by another trained and
experienced facilitator.
The fidelity monitoring checklist dated 3/8/2022 was signed on 4/15/2022. There was no
corresponding attendance sheet showing the case manager facilitating a group on 3/8/2022. The
fidelity monitoring checklist dated 3/9/2022 was signed on 4/6/2022. The other fidelity monitoring
checklist dated 3/9/2022 was signed on 3/17/2022.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no identified
corrective actions noted during the monthly internal fidelity checks for facilitators of Impact of
Crime. The clinical director stated when corrective action is needed, the process will be for the
facilitator to meet with the clinical director to review findings from the completed fidelity monitoring
to address concerns and provide support. The facilitator will be re-assessed at the next month’s
fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of April 15, 2021 through April 14, 2022
shows each evaluation has an area called facilitator skill delivering the evidence-based
programming. Two evaluations were provided for staff who facilitated Impact of Crime. The
transition specialist’s evaluation identified the specific SPEP services he was facilitating. The
evaluation mentioned "all of your SPEP groups are on target to meet the specific dosage," but did
not contain any other specific information. A case manager’s evaluation did contain specific
information about the facilitator’s skill in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 13 youth in the
sample, 69% (9 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 13 youth in the
sample, 92% (12 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 5 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 38% (5 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 69 85 81%
This SPEP report evaluates Impact of Crime, an intervention delivered at Duval Academy.
13 youth completed the program during the review period. Of those youth, all 13 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 13 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 9
received at least the recommended weeks of service. Youth in the sample completed between 13
and 19 weeks of service, with an average of 18 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 12 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 22 and 39 hours of
service, with an average of 34 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring the
facilitator adheres to the manual, fidelity monitoring is completed every month for each facilitator,
and each facilitator’s performance evaluation includes discussion of their skill in delivering this
primary service.
Duval Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Duval IOC.pdf

service_type: Impact of Crime


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Living in Balance
SPEP Review Period: April 15, 2021-April 14, 2022
SPEP Review Conducted: May 25, 2022
Florida Department of Juvenile Justice
Report Date: 8/3/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed the program utilizes Living in Balance as a primary service. The documentation shows
there were six staff trained in the primary service: four therapists, the former clinical director, and
current clinical director. The training is a two-hour in-service training delivered by the clinical
director or regional clinical director. The program has a written protocol for the in-service training,
which documents the schedule for the training. There were four therapists who facilitated the
primary service during the annual review period. Each trained staff member can conduct the
monthly internal fidelity checks. The current clinical director, the regional clinical director, the
former clinical director, and two therapists all conducted internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of April 15, 2021 through April 14, 2022
shows the program utilizes Living in Balance as a primary service. The curriculum is offered to
youth who are diagnosed with a substance abuse diagnosis and/or substance abuse is one of the
top three criminogenic needs on the youth's Residential Assessment for Youth. The clinical
director stated all youth in the program receive the intervention. Each manual also includes CDs
for facilitator resources and session handouts. The manuals include facilitation guidelines, specific
instructions for each lesson, and utilizes role plays, video, and discussion. There are a total of
forty-seven (47) lessons for LIB. The sessions are to be sixty (60) to seventy-five (75) minutes in
duration. The program keeps all SPEP documents in a binder labeled "Living in Balance" in the
clinical director’s office. The program provides the service on Tuesdays and Thursdays for one
hour each.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Living in Balance was observed on 5/24/2022 via Microsoft Teams. There were eight
youth, two staff, and the facilitator present for group. The group began at 12:08pm and was held
in the multi-purpose room. The facilitator was knowledgeable of the material and presented the
material in a way the youth were able to understand. The youth would try to have side
conversations, but the facilitator was able to quickly redirect the youth. The group ended at 1:08p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 15, 2021 showed the program
experienced facilitator turnover which adversely affected the delivery of Living in Balance (LIB). A
therapist left the program at the end of October 2021 and another therapist left at the end of
November 2021. Both therapists were facilitating LIB at the time they left. The program
transitioned the youth to a new trained therapist. There was a two-week gap in service from
6/29/2021-7/13/2021. The primary service is an open group run year-round. According to the
former clinical director, the therapist is assigned to either A-MOD or B-MOD. The clinical director
stated if the program’s census is under fifteen (15) youth, the therapists will take turns facilitating
the primary service. Once the program’s census goes above fifteen (15) youth, each therapist will
facilitate the primary service with the youth in their assigned MOD.
The attendance sheets dated on 12/7/2021, 1/26/2022, and 1/11/2022 had two different cohorts
of youth receiving the primary service at the same time. There were more than fifteen (15) youth
combined receiving the service with those two groups.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed there were twenty-one (21) internal fidelity checks completed for facilitators of Living in
Balance. Four staff facilitated groups during the annual review period. The program was a missing
fidelity check for the month of November 2021 for one therapist and the month of April 2022 for
another therapist. The therapist who was missing the April 2022 fidelity left the program prior to
the fidelity check being completed. The staff who conducted the internal fidelity check for the
annual review period were the current clinical director, the regional clinical director, the former
clinical director, and two therapists. The clinical director stated the program conducts internal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
fidelity monitoring on a monthly basis. Facilitators trained must be monitored monthly by another
trained and experienced facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were identified corrective
actions noted during the monthly internal fidelity checks for facilitators of Living in Balance. The
fidelity check dated for 3/29/2022 stated the facilitator needs to be retrained. The internal fidelity
monitoring was discussed with the therapist during their supervision meeting on 3/31/2022 and
was retrained on 4/4/2022. The fidelity check dated 3/31/2022 stated the facilitator needed to
have more than one fidelity check completed per month. The clinical director stated the therapist
who needed increased monitoring left the program at the end of April 2022.
The clinical director stated when corrective action is needed the process will be for the facilitator
to meet with the clinical director to review findings from the completed fidelity monitoring to
address concerns and provide support. The facilitator will be re-assessed at the next month’s
fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of April 15, 2021 through April 14, 2022
shows each evaluation has an area called facilitator skill delivering the evidence-based
programming. One evaluation was provided for a staff who facilitated Living in Balance. The
therapist’s evaluation did contain specific information about the facilitator’s skill in delivering the
primary service. The remaining therapists were ineligible for an annual evaluation at the time of
this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 13 youth in the
sample, 100% (13 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 13 youth in the
sample, 92% (12 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 5 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 38% (5 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 93 95 98%
This SPEP report evaluates Living in Balance, an intervention delivered at Duval Academy.
13 youth completed the program during the review period. Of those youth, all 13 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 13 total youth sampled, all
13 included dosage with end dates in the EBS Module. Of those youth, 13 received at least the
recommended weeks of service. Youth in the sample completed between 25 and 43 weeks of
service, with an average of 33 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 12 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 35 and 81 hours of
service, with an average of 56 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring facilitator
turnover does not impact groups and fidelity monitoring is completed every month for each
facilitator.
Duval Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 Duval LIB.pdf