summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
640 Roberts Road, Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): March 7 ‐ 10, 2017
Florida Department of Juvenile Justice
Report Date(s): 11/30/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): March 7 - 10, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers 9 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Currently there is one full-time staff member trained and facilitating IOC. There is also a part-time
staff member trained in IOC, however this staff member is no longer facilitating IOC and does not
complete internal fidelity monitoring. The staff members that facilitated IOC during the review
period completed formal facilitator training.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate manual for Impact of Crime (IOC). There are four sections
to the IOC manual. The first section is the facilitator’s guide, which includes an introduction and
implementation section. The second section has all seven chapters divided and numbered. Within
each chapter contains the IOC script, lessons, and activities for the youth. Every chapter begins
with objectives, words to know, and definitions. The third section contains extra worksheets, list
for videos, behavior contracts, and other helpful worksheets to use during group. The final section
is the student guide. This is the specific workbook for the youth. These can be copied and
provided to the youth during group as homework and the worksheets they use during group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime (IOC) was observed during the review. There were nine youth in the
group. The program regularly facilitates IOC between 5:30pm-6:30pm. During the week of the
review, the group was held from 1:00pm-2:00pm. The facilitator utilized the manual and followed
the script and was able to put the lesson into their own words.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The group started with a review from the lesson on Monday. The lesson observed was Chapter 6,
Session 4. The youth displayed the role play appropriately. The facilitator explained the rules two
times for the youth. The group was held outside and the youth had a nice amount of room to
display the role play. The facilitator did a good job to maintain control of the group. The youth
interacted well and were incorporating some of the skills they had learned from different groups.
It was discussed that a co-facilitator would be beneficial and help with the group. This was also
discussed with the designated mental health clinical authority. The plan is to change the schedule
of IOC to later in the morning or around the lunch break and have a co-facilitator.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The facility provided twelve months of records for Impact of Crime (IOC) to review. The facility
maintains a binder for IOC. The binder contains all the attendance sheets. The sheets show the
session dates and time, session length, facilitator names and titles, all of the youth names, JJIS
IDs, and youth signatures. The majority of the attendance sheets contained all youth signatures. If
a youth was absent, the majority of the attendance sheets had a reason listed why the youth was
absent. There was also a section at the bottom that included any youth that were dropped with a
reason why. There was also a section for facilitators to sign and date the attendance sheets. All of
the attendance sheets had signatures completed. The binder was very organized and easy to find
all information needed to complete the review.
The current schedule for IOC at the program from 5:30pm to 6:30pm on Mondays and
Wednesdays.This curriculum is a closed group, meaning once the group has started, no youth
may be added to the group until the next cohort is started.
The program has conducted four cohorts of IOC during the review period. The cohort dates are as
follows: February 2016 to May 2016, May 2016 to September 2016, August 2016 to October
2016, and December 2016 to February 2017. There has been some facilitator turnover during the
review period but this did not impact groups and occurred in between cohorts. There was also a
gap in service. The gap in service occurred between October 31, 2016 and December 12, 2016.
This gap is service was not due to facilitator turnover. The designated mental health clinical
authority (DMHCA) reported that the gap was due to waiting for more youth to enter the facility to
start a new cohort.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring was completed in August 2016, September, 2016 and January 2017.
Internal fidelity monitoring forms were missing for March 2016, April 2016. May 2016, June 2016,
July 2016, October 2016, November 2016, and December 2016. An interview was held with the
designated mental health clinical authority (DMHCA) and it was verified that these internal fidelity
monitorings are missing and were not completed due to lack of staff at the facility being trained in
Impact of Crime (IOC).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective actions on internal fidelity monitoring. The designated
mental health clinical authority (DMHCA) reported that if a corrective action is noted on an internal
fidelity form it is discussed during weekly supervision meetings.
On the internal fidelity monitoring forms completed, no corrective actions were noted. There are
positive comments made regarding the current facilitator.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The current Impact of Crime (IOC) facilitator has not been employed at the facility long enough to
have received a yearly performance evaluation.
Two of the previous IOC facilitators are no longer employed at the facility, but one of them had
received a yearly performance evaluation. The yearly performance evaluation was completed May
27, 2016. One area of the yearly performance evaluation dealt with group facilitation for this
employee. The employee received satisfactory ratings for IOC and there were no further
comments. One of the IOC facilitators is still employed at the facility, but has moved to a part-time
position and no longer facilitates IOC.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 19 youth in the
sample, 5% (1 of 19) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 19 youth in the
sample, 84% (16 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 9 youth
High = 5 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 26%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 5 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (17 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 26% (5 of 19) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 18 25 72%
Totals 66 85 78%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Escambia Boys
Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. None received at least the
recommend weeks of service, with sample youth completing an average of 14 weeks of service.
The program earned 8 points for Amount of Service: Contact Hours. 84% received at least the
recommended hours of service, with sample youth completing an average of 23 hours of service.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score if fidelity
monitoring is completed every month for each facilitator. It may be helpful to have additional staff
within the program who are trained to deliver IOC in order to assist with fidelity monitoring and
minimize the potential for facilitator turnover and gaps in service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

facility_name: Escambia Boys Base

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
640 Roberts Road, Building 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 2/6/18-2/9/18
Florida Department of Juvenile Justice
Report Date(s): 2/8/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): 2/6/18-2/9/18 Lead Reviewer Code: XXXX
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation reflected that five staff were trained in Aggression Replacement
Training (ART) during the review period. The five staff had received formal ART training and
provided documentation to verify the information. ART is co-facilitated at this facility. The
facilitators received formal ART training at Associated Marine Institute (AMI) Kids headquarters
located in Tampa, Florida. The following are dates the staff were trained: 7/8/16, 7/20/16, 4/21/17,
and two were trained on 10/27/17. There are two facilitators currently running ART at this time.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
This program provided the Aggression Replacement Training (ART) manual for this reviewer. This
manual was the third edition-revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. The book designates what the skill is for each lesson, lists the
objectives that should be met, materials that are needed, and then leads into the lesson. There is
a general script to follow and a procedure for each lesson to use as a guide. The skill steps are
broken into sub-steps for the facilitator to follow and steps the youth needs to complete. There are
no student manuals for ART.
The facility maintains a black binder with an orange paper in front with the title “ART”. The
binder was separated into five sections. The first section contains the schedule of when groups
are currently held. ART is currently held on Mondays, Wednesdays, and Fridays from 2:00pm-
3:00pm. The second section contains all the attendance sheets separated by a blank sheet of
green paper per cohort. The attendance sheets are those created by the Department of Juvenile
Justice and include the program name, group sessions date, group session time, facilitator one
and two. The second section includes all of the lessons for ART with checkboxes to indicate what
lesson is being held on the date. The third section has all youth names and JJIS ID numbers
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
typed, a space for the youth signature, and checkboxes to mark if homework was completed. The
last section is to indicate if a youth was dropped and a reason why, number of youth attending
group, and if there were any supportive lessons. The last section is for facilitators to sign and
date. The majority of the attendance sheets contained all youth signatures. If a youth was absent,
the majority of the attendance sheets had a reason listed why the youth was absent. There was
also a section at the bottom that included any youth that were dropped with a reason why. Once a
youth was dropped, that list remained on the attendance sheet until the end of the cohort. There
was also a section for facilitators to sign and date the attendance sheets. All of the attendance
sheets had signatures completed.
The third section includes the internal fidelity monitoring sheets for the full review period. There
were internal fidelity sheets missing during the review period. The fourth section contains the
internal Associated Marine Institute (AMI) fidelity monitoring documents that are created to send
to AMI headquarters on a monthly basis. These are a snapshot of the group for each month and
the program sends this report to AMI headquarters. These reports include, sessions per month,
primary facilitator and training date, youth enrolled at the beginning of the month and number of
youth enrolled at the close of the month, along with an explanation for any enrollment changes.
The next section of the report contains information for each indicator of SPEP, i.e., Facilitator
Training, Primary Service Manual, Observed adherence to Primary Service Manual, Facilitator
Turnover, Internal Fidelity, Corrective Actions, and Evaluation of Facilitator Skill. Each of these
sections has a scoring and comment section for each indicator. The last section of the binder had
certifications for the facilitators. The section contained education information and training
certificates for ART facilitators.
The binder was very organized and easy to find all information needed to complete the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program has held five cohorts of Aggression Replacement Training (ART) during the review
period; March 2017 to May 2017, May 2017 to July 2017, June 2017 to September 2017,
September 2017 to November 2017, and November 2017 to February 2018. A new cohort was
started January 2018. The previous facilitator of that group left during the last cohort. The two
current facilitators took over that group. The cohorts are ran concurrently.
This reviewer observed a group Wednesday, February 7, 2018. There are two facilitators for this
cohort who are both formally trained in ART. The session lasted a full hour and the group
completed lesson 20, the angry behavior cycle. One facilitator explained to this reviewer that the
youth were struggling with some of the concepts, so the current facilitators moved the youth back
five lessons to ensure that the concepts are being understood and demonstrated. The facilitators
provided this reviewer with a copy of the attendance sheet for today’s lesson and the fidelity
monitoring checklist that is provided by the Department of Juvenile Justice. There were ten youth
in today’s group. The youth were very involved in the lesson on today’s date. The facilitators
incorporated youth in assisting with the lesson by writing on the board. All of the youth conducted
role plays on the anger control cycle. The youth were able to pick role-play partners and had
ample time to come up with their role plays. The actors, participants, and facilitators all gave
feedback regarding the role plays. The facilitators handed out homework for the next lesson on
Friday.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Both facilitators appeared very comfortable with the lesson and were able to teach the lesson
without reading from the manual. The facilitators covered all of the objectives for this lesson and
were able to put the lesson in their own words. The facilitators were enthusiastic and positive
during the group. This helped to keep the youth motivated in the discussions. The group
reviewed group expectations and rules at the beginning of the lesson. The facilitators took time to
explain and answer any questions regarding the homework.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Over the review period three different staff facilitated Aggression Replacement Training (ART). Of
the three facilitators, one left employment at the facility during an ART cohort. The two other
facilitators took over that group and continues to facilitate it.
There were no gaps in service during the review period.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program’s process for internal fidelity monitoring is for peers to monitor each other. The
fidelity monitoring sheets are reviewed by the designated mental health clinical authority
(DMHCA) on a weekly basis during supervision time. There are four missing internal fidelity
monitoring forms missing out of the twelve month review period.
Associated Marine Institute (AMI) kids also creates monthly fidelity monitoring reports. This
reviewer spoke with the designated mental health clinical authority (DMHCA) regarding the
process for the internal reports. The DMHCA reported that the Aggression Replacement Training
(ART) fidelity forms are provided to for review. The DMHCA then inputs all information in a report
to send to AMI Headquarters located in Tampa, Florida. AMI Headquarters will then send a full
report back to the program. All twelve AMI Headquarter reports were reviewed.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program process for corrective action based upon internal fidelity monitoring is for the
designated mental health clinical authority (DMHCA) to review all internal fidelity forms and
address any corrective actions during weekly supervisions with the staff facilitating Aggression
Replacement Training (ART). The internal fidelity form used was created by the developer of the
program and is being used by the program. There were no indications of correction action for any
facilitators during the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Two of the five staff have been employed long enough in their positions to have received a yearly
performance evaluation. There is a section labeled group facilitation on the yearly performance
evaluation and it is split into three sections for evaluation: group facilitation skills, group
documentation, and knowledge of evidenced based interventions.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 15 youth in the
sample, 60% (9 of 15) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 15 youth in the
sample, 100% (15 of 15) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 60%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 4 youth
Moderate-High = 3 youth
High = 2 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 13%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 2 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 60% (9 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 13% (2 of 15) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 5 25 20%
Totals 76 100 76%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 15 total youth sampled, all
15 included dosage with end dates in the EBS Module. Of those youth, 9 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 11 weeks of
service, with an average of 10 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, all 15 included dosage in the EBS Module. Of those youth, all 15 received at least the
recommended hours of service. Youth in the sample completed between 30 and 33 hours of
service, with an average of 30 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

source: DJJ

report_type: SPEP

program_name_raw: Escambia Boys Base

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
640 Roberts Road, Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment - MBT/CBT
SPEP Review Date(s): 2/6/18-2/9/18
Florida Department of Juvenile Justice
Report Date(s): 2/12/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): 2/6/18-2/9/18 Lead Reviewer Code: XXXX
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment. The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation reflected that staff is trained through Associated Marine Institute
(AMI) kids headquarters located in Tampa, Florida, conducts Cannabis Youth Treatment (CYT)
formal training. When the training is offered, the designated mental health clinical authority
(DMHCA), sends staff to be trained. Currently, there are two clinical staff members trained in
CYT.
The program's documentation reflected that one therapist was trained in November 2015 and a
second staff member was trained September 2016. A third therapist was formally trained in CYT
in April 2017, but is no longer employed at the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate Cannabis Youth Treatment (CYT) manual. CYT is
designed for youth ages 12-18 with a diagnosis of cannabis abuse or dependence. The youth
must be experiencing emotional, physical, legal, social, and/or academic problems associated
with marijuana use. The manual was copied, put in a binder with a copy of the manual in the front
sleeve, and provided to the staff during formal training. The program facilitates a combination of
the curriculum; Motivational Enhancement Therapy/Cognitive Behavioral Therapy (MET/CBT5
and CBT7). This combination is allowed per the instructor’s manual. The manual includes the
following: implementation information, facilitator requirements, session requirements, script, and
worksheets for the lessons being provided.
The facility maintains a black binder with a paper in front with the title MET/CBT printed on it.
The first section contains the schedule of when groups are currently held. CYT is currently held on
Tuesdays from 2:00pm-3:15pm. The second section contains all the attendance sheets
separated by a blank sheet of green paper per cohort. The attendance sheets included the type of
group, location of group, date, time, group topic, facilitator/credentials, youth names and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
signatures, comment section, and facilitator signature and date. If a youth was absent, the
majority of the attendance sheets had a reason listed why the youth was absent. All of the
attendance sheets had signatures completed. The third section includes the internal fidelity
monitoring sheets for the full review period. There were internal fidelity sheets missing during the
review period. The fourth section contains the internal Associated Marine Institute (AMI)
Standardized Program Evaluation Protocol (SPEP) documents that are created to send to AMI
headquarters on a monthly basis. These are a snapshot of the group for each month and the
program sends this report to AMI headquarters. These reports include, sessions per month,
primary facilitator and training date, youth enrolled at the beginning of the month and number of
youth enrolled at the close of the month, along with an explanation for any enrollment changes.
The next section of the report contains information for each indicator of SPEP, i.e., Facilitator
Training, Primary Service Manual, Observed adherence to Primary Service Manual, Facilitator
Turnover, Internal Fidelity, Corrective Actions, and Evaluation of Facilitator Skill. Each of these
sections has a scoring and comment section for each indicator. The last section of the binder had
certifications for the facilitators. The section contained education information and training
certificates for CYT facilitators. The binder was very organized and easy to find all information
needed to complete the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program has completed four full cohorts and is currently facilitating a fifth cohort. The dates of
the cohorts are as follows: January 2017 to March 2017, March 2017 to May 2017, June 2017 to
August 2017, and October 2017 to December 2017. The group is running as scheduled. Cannabis
Youth Treatment (CYT) is a closed group, which means once the group is started, no youth are
able to be added.
This reviewer observed (CYT) group during the review week. Ten youth were in attendance for
this group and it was facilitated by a formally trained staff. This reviewer obtained an attendance
sheet and fidelity monitoring form for this lesson. The attendance sheet was created by
Associated Marine Institute (AMI) kids. It included what type of therapy group, location, date of
group, time, group topic, facilitator, credentials of the facilitator, signatures of the youth, a
comments section, and the facilitator signature.
The lesson being held during the observed group was CBT8: Anger Management. The facilitator
did a great job with the youth and effectively completed the objectives for the lesson. The
facilitator followed the lesson and was able to put the objectives and lesson into their own words
and did not read from the manual. The facilitator consistently used reflections, affirmations, and
open-ended questions. The youth were engaged, participated well, and asked appropriate
questions. The youth showed a good level of respect for this staff member. The support staff what
was in the group with the facilitator did a great job. The staff addressed a youth that was being
disruptive to group, but did it in a way that did not affect the other youth. The interaction was
appropriate and the youth became more attentive.
The youth successfully completed the role plays for this lesson. The facilitator also did a great job
debriefing the role play. The facilitator reviewed the lesson for the day and the youth were able to
discuss, in their own words, what was taught and what they learned.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Homework from the previous lesson was reviewed and collected. New homework was explained
and assigned to the youth. Each youth has a folder for their group homework.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Review of twelve months of documentation and interview with the designated mental health
clinical authority (DMHCA) verified that there was turnover during the review period. The previous
facilitator for Cannabis Youth Treatment (CYT) left the facility in mid-2017. Two other therapists
were trained in CYT and began facilitating group. There was a gap in service between May 11,
2017 to June 29, 2017 and August 31, 2017 to October 2017. This reviewer spoke with the
DMHCA regarding the gaps in services. Per the DMHCA, the gaps were due to awaiting new
youth that fit the criteria for CYT groups to enter the program.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Twelve months of internal fidelity monitoring forms were reviewed for Cannabis Youth Treatment
(CYT). Five out of twelve months of internal fidelity monitoring forms were accounted for and
reviewed. Seven months of internal fidelity monitoring forms were not completed. This reviewer
spoke with the designated mental health clinical authority (DMHCA) regarding the missing forms.
It was reported that there was not a group held during September 2017 due to not having enough
youth to start a group during that month. For the other months it was reported by the DMHCA and
on the program's monthly reports from Associated Marine Institute (AMI) headquarters that staff
that were able to monitor CYT groups were unable to due to completing other groups at this same
time and not being avialable to complete internal fidelity monitoring. This information was also
documented on the fidelity reports sent to the program monthly by the Associated Marine Institute
(AMI) headquarters, located in Tampa, Florida.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The designated mental health clinical authority (DMHCA) completes weekly clinical supervision
with all therapists and clinical case managers. All fidelity monitoring sheets are reviewed by the
DMHCA and any corrective action indicated on the internal fidelity form is discussed during the
weekly supervision meeting. No corrective actions were indicated on the internal fidelity
monitoring forms during the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
One therapist received a yearly performance evaluation on February 2018. This therapist had
taken 12 weeks of Family Medical Leave was not present for her evaluation due date that was
due in October 2017. There is a section on the yearly performance evaluation for group
facilitation.There are three sub-sections in the performance evaluations that address group
facilitation skills, group documentation, and knowledge of evidence based interventions. There
were no concerns are areas of improvement noted.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 13 youth in the
sample, 0% (0 of 13) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 13 youth in the
sample, 0% (0 of 13) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 8 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 32%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 54%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 3 youth
Moderate-High = 2 youth
High = 2 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 15%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 2 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 54% (7 of 13) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 15% (2 of 13) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 8 25 32%
Totals 63 100 63%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Escambia
Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 13 total youth sampled, all
13 included dosage with end dates in the EBS Module. Of those youth, 0 received at least the
recommended weeks of service. Youth in the sample completed between 7 and 9 weeks of
service, with an average of 8 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed 14 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program was awarded 8 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 80%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can maintain their SPEP Quality of Service Delivery score by continuing the
practice in place at the time of this review.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

service_type: Cannabis Youth Treatment


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
640 Roberts Road, Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime
SPEP Review Date(s): 2/6/18-2/9/18
Florida Department of Juvenile Justice
Report Date(s): 2/7/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): 2/6/18-2/9/18 Lead Reviewer Code:
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers 5 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve months of
SPEP records examinations. Currently there are three formaly trained staff in Impact of Crime
(IOC) at the program. One of the three staff consistently facilitates group and was trained March
2017. A second staff member is used as a back-up facilitator and internal fidelity monitor and was
trained October 2013. This staff member is a part-time employee. The third facilitator has not
started running groups at this time and was trained January 2018. This staff member will be
starting a new cohort when the current one ends. There was a fourth staff member that previously
facilitated IOC groups, but this staff is no longer working at the program.
All four staff were formally trained in the IOC curriculum. The program tries to have, at least, three
staff trained in IOC at all times.
Certificates of training for staff currently facilitating IOC groups were located in the SPEP binder
for IOC.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate manual for Impact of Crime (IOC). There are four sections
to the IOC manual.
The manual includes all needed worksheets and facilitator forms to conduct IOC.
The facility maintains a black binder with an orange paper in front with the title “Impact of Crime”.
The binder was separated into five sections.
The first section contains the schedule of when groups are currently held. IOC is currently held on
Mondays and Wednesays from 12:00pm-1:00pm.
The second section contains all the attendance sheets separated by a blank sheet of green paper
per cohort. The attendance sheets are those created by the Department of Juvenile Justice and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
include the program name, group sessions date, group session time, facilitator one and two. The
second section includes all of the lessons for IOC with checkboxes to indicate what lesson is
being held on the date. The third section has all youth names and JJIS ID numbers typed, a
space for the youth signature, and checkboxes to mark if homework was completed. The last
section is to indicate if a youth was dropped and a reason why, number of youth attending group,
and if there were any supportive lessons. The last section is for facilitators to sign and date.
The third section includes the Department of Juvenile Justice internal fidelity monitoring sheets for
the review period.
The fourth section contains the internal Associated Marine Institute (AMI) fidelity monitoring
documents that are created to send to AMI headquarters on a monthly basis. These are a
snapshot of the group for each month and the program sends this report to AMI headquarters.
These reports include, sessions per month, primary facilitator and training date, youth enrolled at
the beginning of the month and number of youth enrolled at the close of the month, along with an
explanation for any enrollment changes. The next section of the report contains information for
each indicator of SPEP, i.e., Facilitator Training, Primary Service Manual, Observed adherence to
Primary Service Manual, Facilitator Turnover, Internal Fidelity, Corrective Actions, and Evaluation
of Facilitator Skill. Each of these sections has a scoring and comment section for each indicator.
The last section of the binder had certifications for the facilitators. The section contained
education information and training certificates for IOC facilitators.
The binder was organized and easy to find all information needed to complete the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The facility provided twelve months of records for Impact of Crime (IOC) to review. This
curriculum is a closed group, meaning once the group has started, no youth may be added to the
group until the next cohort is started.
The program has conducted four cohorts of IOC during the review period. The cohort dates are as
follows: December 2016 to April 2017, March 2017 through June 2017, June 2017 through August
2017, September 2017 through November 2017. The current cohort began December 18, 2017.
Based on Standardized Program Evaluation Protocol (SPEP) standards during these timeframes,
the cohorts were held to fidelity standards. There was some staff turnover during this time,
however the facility had enough staff to continue the cohorts with no gaps in service over 30 days.
This reviewer was able to observe the Impact of Crime (IOC) group on Wednesday, February 7,
2018 from 12:00pm to 1:00pm. There were twelve youth in group and for this cohort. The
facilitator began group by reviewing the individual goals that they set for themselves during the
last meeting. The group began chapter four today, what interferes with good decision making. The
facilitator asked the youth what are some things that have interfered with them making good
decisions. The facilitator went around the room and each youth shared a personal experience.
The facilitator passed out worksheets for the lesson being covered today. The facilitator reviewed
the objectives and words to know for this chapter. The youth went around and gave their own
definitions for each of the words to know.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Many of the youth identified anger as one feeling that interferes with good decision making. the
youth demonstrated skills learned in Aggression Replacement Training to explain their process to
calm down when angry and how it has effected their decision making in the past.
The youth conducted the role-play on peer pressure from the chapter. The youth provided
feedback and the group provided feedback. The youth did a great job with their role-play and
feedback. The group reviewed the questions to debrief the role-play.
There was positive interactions between the facilitator and the youth. The facilitator is consistent
in providing the lessons and material. The facilitator had worksheets ready and demonstrated a
working knowledge of the material presented. The facilitator was comfortable in teaching the
lesson and did not have to rely on the manual during the group. The group completed objective
one today. The facilitator consistently used affirmations, reflections, and open-ended questions
throughout the group. The facilitator creates a positive environment and there is a good rapport
between the facilitator and the youth.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program had one facilitator leave employment in July of 2017. The program's policy regarding
turnover, per an interview with the Designated Mental Health Clinical Authority (DMHCA), is to
have, at least two to three clinical case management staff trained at all times in the event of
someone being out.
There were no gaps in service during the review period.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
This reviewer reviewed twelve months of internal fidelity monitoring reports. The program had an
internal process to conduct internal fidelity monitoring specific to Impact of Crime (IOC); their
process was for each trained facilitator to observe the current facilitator once a month.
Nine out of twelve months of internal fidelity monitoring was completed and provided for review.
The fidelity monitorings were completed on the IOC fidelity monitoring form.
Associated Marine Institute (AMI) kids also creates monthly fidelity monitoring reports. This
reviewer spoke with the designated mental health clinical authority (DMHCA) regarding the
process for the internal reports. The DMHCA reported that the IOC fidelity forms are provided to
for review. The DMHCA then inputs all information in a report to send to AMI Headquarters
located in Tampa. AMI Headquarters will then send a full report back to the program. For the
three months of internal fidelity forms missing, it was noted that there were no staff available to
complete fidelity monitoring during that time due to the monitoring staff being on a different shift
than when group was being held.
All twelve AMI Headquarter reports were reviewed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective actions on internal fidelity monitoring. The designated
mental health clinical authority (DMHCA) reported that if a corrective action is noted on an internal
fidelity form it is discussed during weekly supervisions.
On the internal fidelity monitoring forms no corrective actions were noted. There are positive
comments made regarding the current facilitator.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The current Impact of Crime (IOC) facilitator has not been employed in the clinical case manager
position one year in order to have received a yearly performance evaluation. This same staff
member was not in his previous position for a full year before being promoted to the current
position and has not received a performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 15 youth in the
sample, 93% (14 of 15) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 67%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 5 youth
Moderate-High = 3 youth
High = 2 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 13%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 2 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 67% (10 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 13% (2 of 15) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 5 25 20%
Totals 53 85 62%
This SPEP report evaluates Impact of Crime, an intervention delivered at Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 15 total youth sampled, all
15 included dosage with end dates in the EBS Module. Of those youth, 0 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 15 weeks of
service, with an average of 9 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, all 15 included dosage in the EBS Module. Of those youth, 14 received at least the
recommended hours of service. Youth in the sample completed between 15 and 55 hours of
service, with an average of 48 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc.
(Contract Provider)
Bldg 3180 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 1/29/19-2/1/19
Florida Department of Juvenile Justice
Report Date(s): 2/1/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation reflected that the program had a training protocol and five staff
members received formal training from a qualified trainer to become a group facilitator in
Aggression Replacement Training. The current staff members trained to facilitate are the
designated mental health clinical authority, three therapists, and one recreational therapist. One
therapist and the recreational therapist left the facility during the review period.
The program policy is for all therapy staff to become trained in Aggression Replacement Training.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
This program provided the Aggression Replacement Training (ART) manual for review. This
manual was the third edition, revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. The book designates what the skill is for each lesson, lists the
objectives that should be met, materials that are needed, and then leads into the lesson. There is
a general script to follow and a procedure for each lesson to use as a guide. The skill steps are
broken into sub-steps for the facilitator to follow and steps the youth needs to complete. There are
no student manuals for ART.
The facility maintains a black binder with an orange paper in front with the title “ART”. The
binder was separated into five sections. The first section contains the schedule of when groups
are currently held. ART is currently held on Mondays, Wednesdays, and Fridays from 2:00pm-
3:00pm. The second section contains all the attendance sheets separated by a blank sheet of
green paper per cohort. All attendance sheets for the review period were accounted for and
signed by all youth that were present and the facilitator. The third section includes the internal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
fidelity monitoring sheets for the full review period. There was one internal fidelity sheets missing
during the review period.
The fourth section contains the internal Associated Marine Institute (AMI) fidelity monitoring
documents that are created to send to AMI headquarters on a monthly basis. These are a
snapshot of the group for each month and the program sends this report to AMI headquarters.
These reports include, sessions per month, primary facilitator and training date, youth enrolled at
the beginning of the month and number of youth enrolled at the close of the month, along with an
explanation for any enrollment changes. The next section of the report contains information for
each indicator of SPEP, i.e., Facilitator Training, Primary Service Manual, Observed adherence to
Primary Service Manual, Facilitator Turnover, Internal Fidelity, Corrective Actions, and Evaluation
of Facilitator Skill. Each of these sections has a scoring and comment section for each indicator.
The last section of the binder had certifications for the facilitators. The section contained
education information and training certificates for ART facilitators. The binder was very organized
and easy to find all information needed to complete the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Aggression Replacement Training (ART) was observed on Wednesday, January 30, 2019 from
2:00pm-3:00pm. There were five youth in group. This group started with more youth, however
some of the youth were arrested and not returned to the facility, and some were released as
successful from the facility.
The youth were on their last anger control group. Friday is the last group for ART. The group
reviewed the anger control cycle. The youth took turns talking about being triggered this week and
how they handled it. The facilitator consistently keeps youth involved. For example, the youth
discussed who helped write on the white board during the last meeting and it rotated to the next
youth with no incident. The direct care staff that assisted in youth was excellent. The direct care
staff participated with the youth, was encouraging towards the youth and helped to assist with role
plays. After each role play the group debriefed and provided feedback.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There were two staff who left during the review period; one therapist and the recreational
therapist. There were no gaps in service over one month due to other staff being trained and able
to step in and take over the group. Currently there are three staff trained in Aggression
Replacement Training.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
This reviewer reviewed twelve months of internal fidleity forms located in the Standardized
Program Evaluation Protocol (SPEP) binder. Of the twelve months, one month of internal fidelity
monitoring was missing.
The process for internal fidelity monitoring is for the designated mental health clinical authority
(DMHCA) to conduct monthly internal fidelity monitoring. One of the other therapists will also
conduct fidelity monitoring periodically when the DMHCA is unavailable.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for the internal fidleity monitoring and corrective action is for the
designated mental health clinical authority to meet with the therapy and case management staff
on a weekly basis. If there are any issues identified on the internal fidelity forms it is addressed
with the the facilitator and reviewed during a following group.
There were no corrective actions or issues indicated on the internal fidelity forms.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the performance evaluations for all group faciltiators for Aggression Replacement
Training (ART) found an entry for evaluation of the staff's performance in group facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 46 youth in the
sample, 61% (28 of 46) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 30 contact hours for this type of service. Of the 46 youth in the
sample, 72% (33 of 46) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 14 youth
High = 19 youth
Total Youth in Sample = 46 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 19 youth
Total Youth in Sample = 46 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (38 of 46) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 41% (19 of 46) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 87 100 87%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
50 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Duration. Of the 46 total youth sampled, only
31 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 28
received at least the recommended weeks of service. Youth in the sample completed between 4
and 25 weeks of service, with an average of 12 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 46 total youth
sampled, all 46 included dosage in the EBS Module. Of those youth with correct dosage, 33
received at least the recommended hours of service. Youth in the sample completed between 8
and 60 hours of service, with an average of 30 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 92%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9