summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime
SPEP Review Period: February 1, 2021 - January 31, 2022
SPEP Review Conducted: February 18, 2022
Florida Department of Juvenile Justice
Report Date: 4/12/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. The clinical case managers and
direct care staff are typically trained to facilitate IOC groups. The program had seven trained staff
in the program during the twelve month review period: two shift supervisors, two direct care staff,
two clinical case managers, one activities coordinator, and one transition specialist.
Six staff facilitated IOC groups during the annual review period. Another trained facilitator
conducts internal fidelity monitoring for IOC groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate manual for Impact of Crime (IOC). There are four sections
to the IOC manual. The manual includes a separate workbook for the youth, which includes
worksheets for each chapter. The manual includes all needed worksheets and facilitator forms to
conduct IOC.
IOC groups are held two days a week. The times and days vary depending upon the schedule of
the facilitator. All documentation for IOC groups are in a binder kept in the director of treatment's
office. The binder includes sign-in sheets, training documentation for the facilitators, internal
fidelity forms, and the schedule for each cohort.
All youth who enter the program are eligible to attend IOC groups. Impact of Crime groups are
held as closed groups, meaning once the first lesson has been completed, no youth can be added
to the cohort.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime group was observed on September 22, 2021. There were six youth in the
group, one facilitator, and one staff member. The group was held from 3:00pm to 4:00pm. The
group completed exercises seven and eight in chapter four. There was one youth who did not
participate and was not respectful during the group. The facilitator became defensive with this
youth. The facilitator lectured the youth throughout the lesson and did not go by the lesson in the
manual. The youth were not engaged during the group.
This reviewer suggested utilizing the manual as there are helpful tips throughout the lessons. It
was also suggested to use the white board and create activites for the youth to stay engaged. The
suggestions were not well received by the facilitator. The issues with the group were discussed
with the director of treatment the same day as the observation. There had been other concerns
with this facilitator as well.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process/protocol for turnover in Impact of Crime (IOC) facilitators is for another
trained staff member to continue facilitating the group.
The program did experience turnover during the annual review period. Two clinical case
managers and one direct care staff left employment during the annual review period. Two of the
staff left during cohorts of IOC. IOC groups are typically not co-facilitated, therefore another
trained staff member continued with the group facilitation. One cohort was co-facilitated during the
annual review period.
There were no gaps in service for IOC groups. All sign-in sheets were provided for the annual
review period. There were three cohorts completed during the twelve-month review period.
One cohort started December 9, 2020 and was completed on April 18, 2021. Six youth started the
group and three youth successfully completed the group. Three youth were released prior to
completing the group. According to the director of treament, the faciltiators did not conduct the
lessons in order and had to go back and complete missed sessions. The facilitators were also
completing the sign-in sheets incorrectly.
The second cohort started April 4, 2021 and was completed on July 21, 2021. Five youth started
the group and three youth successfully completed the group. Two youth were released prior to
completing the group successfully. One youth was ordered a short stay due to being low risk. This
youth received one-to-one sessions in order to successfully complete the curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The third cohort started July 24, 2021 and was completed on November 8, 2021. The group
started with nine youth and ended with ten youth. One youth was arrested, one youth was
removed from the group due to behavioral issues, and another youth was transferred out of the
program.This cohort had a facilitator change due to the original facilitator leaving employment
during the cohort. Another cohort was started on August 28, 2021 with four youth. According to
the director of clinical services, "The four youth who began on August 28, 2021 were doing two-
hour sessions with the facilitator to catch up to the first cohort due to low-risk status." The sign-in
sheets for the second cohort had a start date of July 24, 2021. The four youth in this cohort were
not all low-risk status per a review of the Juvenile Justice Information System; one out of the four
were low risk. This group of four were integrated into the initial cohort started on July 24, 2021
once this group had caught up to the lesson of the other group. A further review confirmed this
cohort of four youth should have maintained as a group and completed the curriculum together.
The second cohort joined the first cohort on September 28, 2021.
The fourth cohort started November 22, 2021 with five youth. Another cohort was started on
December 22, 2021 with two youth. The December 22nd cohort joined the November 22nd cohort
on January 10, 2022. The December 22nd cohort conducted three days of two-hour sessions to
"catch up to" the November 22nd cohort. As of January 10, 2022 there were seven youth in this
cohort and it was continuing to meet at the time of this review.
Technical Assistance will be offered to the program regarding SPEP and service fidelity.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process/procedure for internal fidelity monitoring is for another trained facilitator to
conduct observations. The shift supervisor or transition specialist conducts internal fidelity
monitoring observations. There were six staff who conducted Impact of Crime (IOC) groups.
All internal fidelity monitoring forms were provided for review. Seventeen out of seventeen internal
fidelity monitoring observations were completed during the annual review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's corrective action process is for the director of treatment to address any significant
issues or corrective actions with the facilitators for Impact of Crime.
During the first quarter, two facilitators did receive corrective actions. The director of treament
reported two of the facilitators were not providing the lessons in order and were not ensuring the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
sign-in sheets were completed correctly. The director of treatment met with the facilitators,
showed them how to properly mark the sessions and discussed if a session was not completed
they should continue it to the next session to properly complete it. At this time the cohort is back
on track and lessons are being delivered as designed.
Another facilitator was having issues during the second quarterly review. This facilitator met with
the director of treatment. This reviewer met with the director of treatment regarding the concerns
observed during the external fidelity monitoring on September 22, 2021. The executive director
and director of treatment had been meeting with this faciliator to address the issues. This
facilitator left employment during the third quarter.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided three of six annual performance evaluations for facilitators of Impact of
Crime. There is a section for evidence-based interventions, but does not specifically mention
which services on which the facilitator is being evaluated.
Two staff left employment before receiving an annual performance evaluation. One facilitator has
not been employed long enough to have received an annual performance evaluation. One
facilitator received a forty-five day evaluation, but was not employed long enough to receive an
annual evaluation. This facilitator was not evaluated group facilitation due to not conducting
groups during that forty-five day period.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 16 youth in the
sample, 69% (11 of 16) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 16 youth in the
sample, 81% (13 of 16) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 10 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 75%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 19%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 3 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 75% (12 of 16) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 19% (3 of 16) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 10 25 40%
Totals 64 85 75%
This SPEP report evaluates Impact of Crime, an intervention delivered at Escambia Boys Base.
17 youth completed the program during the review period. Of those youth, 16 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 16 total youth sampled, all
16 included dosage with end dates in the EBS Module. Of those youth, 11 received at least the
recommended weeks of service. Youth in the sample completed between 13 and 19 weeks of
service, with an average of 16 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, all 16 included dosage in the EBS Module. Of those youth, 13 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 14 and 37 hours of
service, with an average of 23 hours.
The program was awarded 10 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring the
facilitator adheres to the manual and facilitator turnover does not impact groups
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 EBB IOC.pdf


state: FL

facility_name: Escambia Boys Base

summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Period: February 1, 2022 - January 31, 2023
SPEP Review Conducted: February 21, 2023
Florida Department of Juvenile Justice
Report Date: 7/27/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Aggression Replacement Training (ART) as a primary service for youth who
have specific identified risk factors on the Residential Assessment for Youth. There is a training
protocol in place for facilitators of ART. There were ten (10) total staff trained during the annual
review period: two clinical directors, corporate staff, director of operations, three therapists, two
transition specialists, and a clinical case manager. The clinical director conducts internal fidelity
monitoring for the ART groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual the program utilizes is the Aggression Replacement Training (ART) third edition-
revised and expanded by authors Barry Glick and John Gibbs. The manual is organized by each
lesson: skill streaming training, moral reasoning training, and anger control training. Each lesson
is divided by weeks. The book designates what the skill is for each lesson, lists the objectives that
should be met, materials needed, and then leads into the lesson. There is a general script to
follow and a procedure for each lesson to use as a guide. The skill steps are broken into sub-
steps for the facilitator to follow and steps the youth needs to complete. There are no student
manuals for ART.
Youth are added to ART groups based on identified risk factors on the Residential Assessment for
Youth, including aggression, attitudes/behaviors, and skills. ART groups must be co-facilitated
with the primary facilitator being a qualified professional defined in F.A.C. 63N-1.0031.
ART groups are held on Mondays, Wednesdays, and Fridays from 2:00pm to 3:00pm. The
program maintains records in a binder kept in the clinical director's office. The binder includes all
sign-in sheets, internal fidelity forms, and all training and qualification records for the facilitators.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Aggression Replacement Training is held as a closed group, meaning no youth may be added
after the first lesson has been completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Aggression Replacement Training (ART) group was observed on Tuesday, November 1, 2022.
There were two facilitators, one staff, and ten (10) youth in attendance. The lesson for the day
was Skillstreaming week 4: Dealing with someone else's anger. Homework was reviewed from
the previous session. The facilitator conducted the lesson in accordance with the manual, but
used his own words. All ten (10) youth were engaged throughout the group. The facilitator and co-
facilitator did not conduct a role play. However, the new skill steps were reviewed and
demonstrated by the primary facilitator for the group today. This reviewer discussed the omission
of the role play with the facilitators. They reported they did not have time to prepare for the role
play prior to today's group. It was discussed the facilitators needed to ensure to have a role-play
for the youth so the expectations are clear. Motivational Interviewing skills (affirmations,
reflections, and open-ended questions) were used throughout the group by the facilitator. All ten
(10) youth conducted a role play demonstrating the steps of the new skill. The youth participated
in providing feedback for the role plays.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of Aggression Replacement Training (ART) facilitator turnover
is for the co-facilitator to continue with group facilitation. Four trained ART facilitators left
employment during the annual review period. Two staff left between cohorts and one staff left
during a cohort. The staff leaving did not effect gaps in service for ART.
The program completed four cohorts of ART during the annual review period. The first cohort
began January 24, 2022 and ended on April 7, 2022. The group started with four youth and three
youth successfully completed the group. One youth was released prior to successfully completing
the cohort.
The second cohort was started May 9, 2022 and ended on July 29, 2022. The group began with
ten (10) youth and ten (10) youth successfully completed the cohort.
The third cohort began July 3, 2022 and ended on September 16, 2022. The group began with
ten (10) youth and ten (10) youth successfully completed the cohort.
The fourth cohort began on October 10, 2022 and ended on December 22, 2022. The group
began with nine youth and ten (10) youth successfully completed the group. One youth was
added in lesson three, with permission from the department. This youth had individual sessions
for lessons one and two and those sign-in sheets were provided for review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A fifth cohort began on January 30, 2023 and was continuing to meet at the time of the review.
The group began with ten (10) youth and there were still ten (10) youth in the group at the time of
the review.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the clinical director to conduct internal fidelity monitoring
observations for the facilitators or Aggression Replacement Training (ART) and for each facilitator
to be monitored three times during each month of facilitation. There were five trained staff who
facilitated ART groups during the annual review period. All internal fidelity forms were provided for
review. There were no missing internal fidelity observations for ART during the annual review
period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There is a process in place to address corrective actions identified during monthly internal fidelity
monitoring. The clinical director addresses any significant issues or corrective actions for the
facilitators of Aggression Replacement Training (ART).
There were no corrective actions for the facilitators of ART during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The clinical director was the only facilitator for Aggression Replacement Training (ART) groups
who had been employed for a year or longer.
The clinical director's performance evaluation was completed on August 26, 2022 and included
documentation of services trained in and an evaluation of facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 31 youth in the
sample, 100% (31 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 31 youth in the
sample, 97% (30 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 71%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 5 youth
Moderate-High = 8 youth
High = 9 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 29%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 9 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 71% (22 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 29% (9 of 31) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 13 25 52%
Totals 86 100 86%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
31 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Duration. Of the 31 total youth sampled, all
31 included dosage with end dates in the EBS Module. Of those youth, 31 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 24 weeks of
service, with an average of 14 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, all 31 included dosage in the EBS Module. Of those youth, 30 received at least the
recommended hours of service. Youth in the sample completed between 25 and 60 hours of
service, with an average of 37 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 98%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2022-23 EBB ART.pdf

program_name_raw: Escambia Boys Base

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment
SPEP Review Period: February 1, 2022 - January 31, 2023
SPEP Review Conducted: February 21, 2023
Florida Department of Juvenile Justice
Report Date: 7/28/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Cannabis Youth Treatment (CYT) as a primary service for youth with
substance abuse diagnosis. The program's protocol for facilitators is for a master's level therapist
to conduct the three individual sessions for each youth. A bachelor's level therapist may conduct
the group sessions.
There were nine staff trained to facilitate CYT groups during the annual review period: two clinical
directors, four therapists, two clinical case managers, and one corporate staff. All staff met the
requirements for facilitation. All training records and qualifications were provided for review and
are maintained in the Standardized Program Evaluation Protocol (SPEP) binders kept in the
clinical director's office.
There were two facilitators for CYT groups during the annual review period. The clinical directors
conducted internal fidelity monitoring for the facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate Cannabis Youth Treatment (CYT) manual. The manual
was copied, put in a binder with a copy of the manual in the front sleeve, and provided to the staff
during formal training. The program facilitates a combination of the curriculum; Motivational
Enhancement Therapy/Cognitive Behavioral Therapy (MET/CBT5 and CBT7). This combination is
allowed in accordance with the instructor’s manual. The curriculum starts with three individual
sessions with a trained therapist before creating the group. The manual includes the following:
implementation information, facilitator requirements, session requirements, script, and worksheets
for the lessons being provided. This reviewer verified with the director of treatment the program is
still using this manual and it has not changed during the twelve (12)-month review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Cannabis Youth Treatment is designed for youth ages twelve (12) to eighteen (18) with a
diagnosis of cannabis abuse or dependence. The youth must be experiencing emotional,
physical, legal, social, and/or academic problems associated with marijuana use.
Cannabis Youth Treatment (CYT) groups are held on Tuesdays from 1:00pm to 2:15pm each
week. All documentation for CYT is maintained in a binder kept in the clinical director's office and
include all sign-in sheets, internal fidelity observations, and training/qualification records for all
facilitators. Cannabis Youth Treatment groups are closed groups, meaning no youth may enter
the group once the initial group lesson has been completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A virtual observation of Cannabis Youth Treatment was held on Friday, January 13, 2023 from
2:00pm to 3:15pm. There were nine youth present, one facilitator, and one direct care staff. The
lesson the youth were completing was anger triggers. The facilitator reviewed group rules and
conducted a check-in. Homework from the previous lesson was reviewed as a whole group. The
facilitator kept the group on topic throughout the lesson. The facilitator was well prepared for the
lesson having the activities, worksheets, pencils, and folders ready for the group. The facilitator
ensured the youth understood the directions for the lesson activities and assigned homework for
the next session.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of staff turnover for Cannabis Youth Treatment (CYT) groups
is for another trained staff to continue with the cohort. The program experienced facilitator
turnover during the annual review period. Two therapists and one corporate staff left employment;
however, the staff were not facilitating CYT groups at the time they left and therefore it did not
effect the youth or cause gaps in service.
The program completed two cohorts during the annual review period. The first cohort started
March 8, 2022 and ended on April 30, 2022. The group began with four youth and three youth
successfully completed the group. One youth was released from the program prior to successfully
completing the cohort.
The second cohort started June 28, 2022 and ended on August 28, 2022. The group began with
ten (10) youth and ten (10) youth successfully completed the cohort.
A third cohort began on December 30, 2022 and was still being held at the time of the review.
Nine youth started the group and nine youth continued in the group at the time of the review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the clinical director to conduct internal fidelity observations for
Cannabis Youth Treatment (CYT) groups. There were two main facilitators for the CYT groups
during the annual review period. A review of all internal fidelity forms confirmed no observations
were missing for the review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process to address corrective actions identified during internal fidelity
monitoring. The program's protocol is for the clinical director to address any signficant issues or
corrective actions for the facilitators of Cannabis Youth Treatment (CYT).
There were no corrective actions for CYT noted nor taken during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The clinical director was the main facilitator for Cannabis Youth Treatment (CYT) during this
annual review period. The other trained staff did not conduct groups and/or have not been
employed for more than a year at the time of the annual review.
The clinical director's performance evaluation was completed on August 26, 2022 and included
documentation of services trained in and an evaluation of facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 12 weeks for this type of service. Of the 15 youth in the
sample, 13% (2 of 15) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 12 contact hours for this type of service. Of the 15 youth in the
sample, 93% (14 of 15) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 6 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (12 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 40% (6 of 15) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 83 100 83%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Escambia
Boys Base.
31 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Duration. Of the 15 total youth sampled, only
14 included dosage with end dates in the EBS Module. Of those youth, 2 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 20 weeks of
service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, all 15 included dosage in the EBS Module. Of those youth, 14 received at least the
recommended hours of service. Youth in the sample completed between 1 and 22 hours of
service, with an average of 12 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 EBB CYT.pdf

service_type: Cannabis Youth Treatment


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime
SPEP Review Period: February 1, 2022 - January 31, 2023
SPEP Review Conducted: February 21, 2023
Florida Department of Juvenile Justice
Report Date: 7/28/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service for all youth in the program. The
program's protocol is for direct care staff and/or clinical case management staff to have formal
training and facilitate IOC groups. During the annual review period, six staff were formally trained
in IOC and all training records were provided for review.
There were three main facilitators during the annual review period. The director of operations
conducts internal fidelity monitoring for the facilitators each month.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the current Impact of Crime (IOC) manual in the program. There are four
sections to the IOC manual and this includes a separate workbook for the youth. The manual
includes all worksheets and facilitator forms to conduct IOC.
Impact of Crime groups are held two days a week for one hour. The days of the week vary based
upon the work schedule of the facilitator. The program maintains binders to include the separate
cohorts conducted or being conducted, internal fidelity forms, and all training documentation. The
IOC groups are closed, meaning no youth may be added to the group once the first lesson has
been completed.
There have been significant issues with the groups being held during the weekends and after
hours. This reviewer spoke with the clinical director and director of operations regarding the issue.
The plan is to move the groups back to the weekdays where there can be better oversight for the
cohorts.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of an Impact of Crime group was conducted on September 22, 2022 from 1:00pm
to 2:00pm. In attendance was one facilitator and six youth. The lesson was chapter 5, objective 2.
The facilitator was comfortable with conducting the group. The facilitator had a pre-approved
video for the youth to watch. This video sparked an open discussion with the youth and the
facilitator. The group completed a worksheet from the youth workbooks and reviewed in class.
There were no behavioral issues during the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of Impact of Crime (IOC) facilitator turnover is for another
trained staff to continue with group facilitation. The groups are not co-facilitated at the program.
The program did experience turnover during the annual review period. The recreational therapist
left employment in August 2022. The facilitator left between cohorts and therefore it did not impact
group facilitation. There were no gaps in service during the annual review period.
All sign-in sheets were provided for review. The program completed four cohorts during the
annual review period. The first cohort started January 10, 2022 and ended on March 23, 2022.
Seven youth started the group and seven youth successfully completed the group.
A second cohort started April 22, 2022 and ended August 17, 2022. The cohort started with eight
youth and eight youth successfully completed the group. The sign-in sheets were missing start
and end times of the groups. This issue was addressed with the facilitator and clinical director.
A third cohort began May 24, 2022 and ended on November 13, 2022. Five youth started the
group and six youth completed the group. One youth was added to the cohort at the beginning of
chapter three. This youth received individual sessions and those sign-in sheets were reviewed.
Two youth were added during chapter six. This reviewer spoke with the clinical director and
director of operations regarding this issue. After interviewing the main facilitator and the facilitator
who assisted with group facilitation at the time the youth were added, there were no clear answers
provided as to why these youth were added to the cohort. It was reiterated to the staff no youth
should be added to a cohort after the first lesson is completed.
The facilitator of this cohort has struggled to meet the requirements of properly facilitating an IOC
group. The group was not held consistently throughout its duration. There have been gaps with
group meeting anywhere from one week to three weeks. At the time of this review, there were ten
(10) sessions left for this cohort. These issues were discussed with the clinical director and the
director of operations. The facilitator has previously been provided peer training, one-on-one
meetings on three different occassions, the group schedule was posted throughout the facility as
a daily reminder, and the clinical director has met with the shift supervisor to work on a schedule
to better fit with group time. The facilitator completed the cohort, however will not be continuing to
facilitate groups in the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A fourth cohort began September 28, 2022 and ended on January 15, 2023. Eleven (11) youth
started the group and eight youth successfully completed the group. The groups are held on
Saturdays and Sundays, but the times were not consistent on the sign-in sheets. One youth was
released early, however there was documentation the youth received individual sessions to
complete the curriculum. Three youth were released from the program prior to successfully
completing the IOC curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process for Impact of Crime (IOC) internal fidelity monitoring is for the director of
operations or another trained staff to conduct observations. There were three facilitators during
the annual review period. There were no missing internal fideilty observations during the annual
review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There is a process in place to address corrective actions identified during internal fidelity
monitoring. The program's policy is for the director of operations and/or clinical director to address
significant issues or corrective actions with the facilitators for Impact of Crime (IOC).
There was one corrective action, mentioned in narrative (d). This issue has been resolved.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
There was one facilitator for Impact of Crime groups who had been employed long enough to
have an annual performance evaluation. This evaluation was dated July 6, 2022.
A review of the annual performance evaluation showed there was no specific section to address
group facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 31 youth in the
sample, 58% (18 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 31 youth in the
sample, 81% (25 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 71%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 5 youth
Moderate-High = 8 youth
High = 9 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 29%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 9 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 71% (22 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 29% (9 of 31) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 13 25 52%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Escambia Boys Base.
31 youth completed the program during the review period. Of those youth, all 31 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 31 total youth sampled, only
27 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 18
received at least the recommended weeks of service. Youth in the sample completed between 11
and 26 weeks of service, with an average of 18 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 27 included dosage in the EBS Module. Of those youth with correct dosage, 25
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended hours of service. Youth in the sample completed between 22
and 35 hours of service, with an average of 26 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 EBB IOC.pdf

service_type: Impact of Crime