summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMI Kids, Inc
(Contract Provider)
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Date(s): May 10‐13, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base QI Program Code: 1271
Provider Name: AMI Kids, Inc Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): May 10-13, 2016 Lead Reviewer Code: 127
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 5 # Case Managers 3 # Youth
DHA or designee 2 # Clinical Staff 5 # Other (listed by title):
DMHCA or designee 3 # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An interview with the Designated Mental Health Authority (DMHA) confirmed five staff members
are trained in Aggression Replacement Training (ART) for the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program currently uses the Aggression Replacement Training (ART) manual which outlines
the details of the curriculum. The ART groups are held every Monday, Wednesday, and Friday at
2:00 PM CST.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An ART group was observed prior to the annual compliance review. The group began with youth
checking in with the group using a comparison to themselves and the weather. A white board and
easel were used to write youth answers down during group discussion. The youth were divided
into groups and performed a role play exercise in front of the group. The facilitator discussed the
exercise asking youth and all group members about coping skills they observed.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There has been no reported facilitator turnoveror gaps in service for this review period.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The Designated Mental Health Authority (DMHA) is a Licensed Clinical Mental Health Counselor
and she has received the necessary training to conduct fidelity monitoring of the facilitators. She
observes each group for fidelity monitoring each month. In addition, the DMHA is required by
AMIkids, Inc. to provide its own monthly observation reports to their headquarters.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Interviews with the Designated Mental Health Authority (DMHA) reported that any fidelity
monitoring observations are noted on the AMIkids checklist form. These observations are
included in the weekly supervision notes. Should the DMHA feel that guidance or coaching is
necessary for the facilitator, that facilitator would be paired up with a stronger facilitator for
guidance. If the issue was not resolved, additional formal training would be sought. Any idenified
issues were applied and demonstrated during this review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The Designated Mental Health Authority (DMHA) reports her evaluations on delivery of services,
are included in the staff’s performance evaluations. Staff receive performance evaluations for the
first 45-days of hire, 90-days of hire, and then every year. A review of the staff evaluations
reviewed during the annual compliance review, revealed that not all staff are receiving their
evaluations according to the program’s policy. In reviewing six staff records, two staff were
missing evaluations. One of the two staff missing evaluations was a clinical staff who facilitates
groups and he had not received an evaluations since October 2014.
The program can earn 1 point if each facilitator is evaulated specifically related to ART.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 10 weeks for this type of service. Of the 19 youth in the
sample, 53% (10 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 19 youth in the
sample, 89% (17 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 8 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (17 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (8 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 90 100 90%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 19 total youth sampled, only
14 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 10
received at least the recommended weeks of service. Youth in the sample completed between 4
and 10 weeks of service, with an average of 9 weeks. The targeted number of weeks for this
specific intervention is 10 weeks. In previous SPEP review years, ART was held to the same
standard as other Group 5 Services (15 weeks); however, according to Dr. Lipsey's SPEP
research, ART meets criteria for an exception to the standard SPEP 15-week dosage target.
Youth are considered to have met the target if they achieve 80% of the dosage specified in the
ART manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled,17 received at least the recommended hours of service. Youth in the sample completed
between 12 and 30 hours of service, with an average of 27 hours. The targeted number of hours
for this type of intervention is 30 hours. In previous SPEP review years, ART was held to the
same standard as other Group 5 Services (45 hours); however, according to Dr. Lipsey's SPEP
research, ART meets criteria for an exception to the standard SPEP 45-hour dosage target. Youth
are considered to have met the target if they achieve 80% of the dosage specified in the ART
manual.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring
perfomance evaluations address delivery of ART.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

report_type: SPEP

program_name_raw: Escambia Boys Base

program_status_note: Open

service_type: Aggression Replacement Training


state: FL

facility_name: Escambia Boys Base

summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMI Kids, Inc
(Contract Provider)
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment
SPEP Review Date(s): May 10‐13, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base QI Program Code: 1271
Provider Name: AMI Kids, Inc Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): May 10-13, 2016 Lead Reviewer Code: 127
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 5 # Case Managers 3 # Youth
DHA or designee 2 # Clinical Staff 5 # Other (listed by title):
DMHCA or designee 3 # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment. The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An interview with the Designated Mental Health Authority (DMHA) confirmed three staff members
are trained to facilitate Cannabis Youth Treatment (CYT) for the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program currently uses the Cannabis Youth Treatment manual which outlines the course of
the curriculum. Any youth admitted with a substance abuse diagnosis and/or any youth diagnosed
with a substance use disorder at the time of their bio-psychosocial evaluation is placed in the next
CYT cohort. CYT groups are held every Tuesday at 2 PM CST.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A CYT group was observed prior to the annual compliance review. The group began with youth
checking in with the group using a comparison to themselves and the weather. The facilitator
used the manual as a guide to facilitate the group. A white board was used to post the agenda for
session and handouts were utilized. The facilitator frequently went around the group to get all
youth to participate. The youth completed an exercise while in their group. Both direct care staff
and the facilitator assisted youth with the exercise and all youth presented their answers. The
facilitator took time to talk youth through their answer and what they thought. At the end of group,
youth turned in their exercise sheets and took homework assignment sheets.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was no reported facilitator turnover for the Cannabis Youth Treatment (CYT) group during
this review period.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The DMHA is a Licensed Clinical Mental Health Counselor and has received the required training
to conduct fidelity monitoring of the facilitators. She has received training in all groups and
observes each group for fidelity monitoring each month. In addition, the DMHA is required by
AMIkids, Inc. to provide its own monthly observation reports to their headquarters.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Interviews DMHA reported that any fidelity monitoring observations are noted on the AMIkids
fidelity monitoring checklist form and observations are included in the weekly supervision notes.
Should the DMHA feel that guidance or coaching is necessary for the facilitator, that facilitator
would be paired up with a stronger facilitator for guidance. If the issue was not resolved the
facilitator would receive additional formal training. If a concern continued to be present, ultimate
action would be to remove the facilitator from facilitating groups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The DMHA reports her evaluations on delivery of services, are included in the staff’s performance
evaluations. Staff receive performance evaluations for the first 45-days of hire, 90-days of hire,
and then every year. A review of the staff evaluations reviewed during the annual compliance
review, revealed that not all staff are receiving their evaluations according to the program’s policy.
In reviewing six staff records, two staff were missing evaluations. One of the two staff missing
evaluations was a clinical staff who facilitates groups and he had not received an evaluations
since October 2014.
The program can earn 1 point if if each facilitator is evaulated specifically related to CYT.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 4 youth
High = 9 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (17 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 50% (9 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 78 100 78%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Escambia
Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 18 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, none
received at least the recommended weeks of service. Youth in the sample completed between 6
and 9 weeks of service, with an average of 9 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 14 and 15 hours of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring staff
evaluations include an assessment of skill in delivering CYT.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

service_type: Cannabis Youth Treatment


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime
SPEP Review Date(s): May 10‐13, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base QI Program Code: 1271
Provider Name: AMI Kids, Inc Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): May 10-13, 2016 Lead Reviewer Code: 127
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 5 # Case Managers 3 # Youth
DHA or designee 2 # Clinical Staff 5 # Other (listed by title):
DMHCA or designee 3 # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
An interview with the Designated Mental Health Authority (DMHA) confirmed two staff members
were trained to facilitate Impact of Crime (IOC) for the program. In the last twelve months, the
Impact of Crime groups have been ongoing. Impact of Crime is provided to all youth in the facility
regardless of their history or specific treatment goals. Per the DMHA, the IOC groups are held
Tuesday and Thursday mornings at 5:30 AM CST.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Impact of Crime manual which outlines the course of the curriculum. The
manual consists of worksheets and detailed lesson plans. Each group session has an
introduction, guided group discussion activity, worksheets, and a conclusion in which the lesson is
summarized.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The IOC group time was altered and observations were completed prior to the annual compliance
review. The group began with youth checking in with the group using a one to ten scale. The
facilitator was observed to be using the manual as a guide to facilitate the group. A white board
was used to post group rules and youth were provided worksheets to complete.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
An interview with the DMHA revealed that in September 2015, a clinical staff member facilitating
the Impact of Crime (IOC) resigned and the group was not completed. In October, a direct care
staff started a new youth group.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The DMHA is a Licensed Clinical Mental Health Counselor and has received the required training
to conduct fidelity monitoring of the facilitators and observes each group for fidelity monitoring
each month. In addition, the DMHA is required by AMIkids, Inc. to provide its own monthly
observation reports to their headquarters.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The DMHA reported all fidelity monitoring observations are noted on the AMIkids fidelity
monitoring checklist form and observations are included in the weekly supervision notes. Should
the DMHA feel that guidance or coaching is necessary for the facilitator, that facilitator would be
paired up with a stronger facilitator for guidance. If the issue was not resolved the facilitator would
receive additional formal training. If a concern continued to be present, ultimate action would be
to remove the facilitator from facilitating groups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The DMHA reports her evaluations on delivery of services, are included in the staff’s performance
evaluations. Staff receive performance evaluations for the first 45-days of hire, 90-days of hire,
and then every year. A review of the staff evaluations revealed that not all staff are receiving their
evaluations according to the program’s policy. In reviewing six staff records, two staff were
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
missing evaluations. One of the two staff missing evaluations was a clinical staff who facilitates
groups and he had not received an evaluations since October 2014.
The program can earn 1 point if each facilitator is evaluated specifically related to IOC.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 16 youth in the
sample, 0% (0 of 16) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 16 youth in the
sample, 56% (9 of 16) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 6 youth
High = 6 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (15 of 16) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 38% (6 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 16 total youth sampled,
none received at least the recommended weeks of service. Youth in the sample completed
between 6 and 15 weeks of service, with an average of 12 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, 9 received at least the recommended hours of service. Youth in the sample completed
between 16 and 24 hours of service, with an average of 21 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by minimizing the
impact of facilitator turnover.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
Building 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training (ART)
SPEP Review Date(s): March 7 ‐ 10, 2017
Florida Department of Juvenile Justice
Report Date(s): 11/30/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): March 7 - 10, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 7 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training (ART). The program
was awarded 30 points because the Primary Service is identified as a Group 5 Service. The
specific Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary
Service was identified as this type of service as it is intended to correct faulty cognitions and
perceptions and provides skills individuals can use to monitor thought patterns and correct
behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had ten different facilitators for Aggression Replacement Training (ART) during the
review period. All ten staff had received formal ART training and provided documentation to verify
the information. ART is co-facilitated at this facility. The facilitators received formal ART training at
AMIkids headquarters located in Tampa, Florida. Of the ten facilitators, four staff are still actively
facilitating ART or completing internal fidelity monitoring for the program at the time of the review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
This program provided the Aggression Replacement Training (ART) manual for review. The
manual was the third edition-revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. There is a general script to follow and a procedure for each lesson.
The skill steps are broken into sub-steps for the facilitator to follow and steps the youth needs to
complete. There are no student manuals for ART.
The facility maintains a binder titled “ART”. The binder contains the schedule of when groups are
currently held. ART is currently held on Mondays, Wednesdays, and Fridays from 2:00pm-
3:00pm. The binder also contains all the attendance sheets. The attendance sheets are those
created by the Department of Juvenile Justice and include the program name, group sessions
date, group session time, facilitators, and youth information. The majority of the attendance
sheets contained all youth signatures. If a youth was absent, the majority of the attendance
sheets had a reason listed why the youth was absent. The binder also included the internal fidelity
monitoring sheets for the full review period and internal AMI SPEP documents that are created to
send to AMI headquarters on a monthly basis. The last section of the binder had certifications for
the facilitators. The section contained education information and training certificates for ART
facilitators.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The binder was very organized and easy to find all information needed to complete the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program has held five cohorts of ART during the review period; February 2016 to April 2016,
February 2016 to May 2016, May 2016 to July 2016, August 2016 to October 2016, and
November 2016 to February 2017. A new cohort was started in March 2017. Two of the cohorts
were run concurrent. A total of eight different facilitators have run ART groups at the facility during
the review period.
An Aggressive Replacement Training (ART) group was observed during the review week. There
are two facilitators for this cohort who are both formally trained in ART. The session lasted a full
hour and the group completed the second lesson, anger control training – Introduction to
ACT/ABC’s of anger. The facilitators provided this reviewer with a copy of the attendance sheet
for today’s lesson and the fidelity monitoring checklist. There were ten youth in the group. The
youth were very involved in the lesson. The facilitators incorporated youth in assisting with the
lesson by writing on the board. Both facilitators appeared very comfortable with the lesson and
were able to teach the lesson without reading from the manual. The facilitators covered all of the
objectives for this lesson and were able to put the lesson in their own words. The facilitators were
enthusiastic and positive during the group. This helped to keep the youth motivated in the
discussions.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Over the review period ten different staff facilitated Aggression Replacement Training (ART). Of
the ten, four staff are still working at the program and facilitating ART. There have been no gaps
in service during the review period. The staff were able to transition to a trained staff member in
the event of a facilitator leaving. Co-facilitation is used, so youth are not impacted by turnover.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program’s process for internal fidelity monitoring is for peers to monitor each other. The
fidelity monitoring sheets are reviewed by the designated mental health clinical authority
(DMHCA) on a weekly basis during supervision time. There were no internal fidelity monitoring
sheets complete during the first cohort (February to April 2016). For the cohort completed
February 2016 to May 2016, only February monitroing was completed. All internal fidelity forms
were completed for the third cohort (May to July 2016). September 2016 and October 2016
internal fidelity monitoring forms were missing for the fourth cohort (completed August to October
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
2016). The fifth cohort had internal fidelity monitoring forms missing for November, December
and January.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for corrective action based upon internal fidelity monitoring is for the
designated mental health clinical authority (DMHCA) to review all internal fidelity forms and
address any corrective actions during weekly supervision meetings with the staff facilitating
Aggression Replacement Training (ART).
Two of the facilitators received low scores on the internal fidelity form during the review period.
The DMHCA monitored the staff three times over the course of a month. The fidelity forms
marked as improved for both of the staff members. The documentation was specific. An interview
with the DMHCA verified the information in the documentation.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Of the ten staff that facilitated Aggression Replacement Training (ART), two staff had received
yearly performance evaluations. The yearly performance evaluations contained a section for
group facilitation where the staff members were rated specifically on ART.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 21 youth in the
sample, 95% (20 of 21) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 21 youth in the
sample, 95% (20 of 21) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 11 youth
High = 5 youth
Total Youth in Sample = 21 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 24%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 5 youth
Total Youth in Sample = 21 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (19 of 21) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 24% (5 of 21) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 15 25 60%
Totals 86 100 86%
This SPEP report evaluates Aggression Replacement Training (ART), an intervention delivered at
Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. 95% received at least the
recommend weeks of service, with sample youth completing an average of 11 weeks of service.
The program earned 8 points for Amount of Service: Contact Hours. 95% received at least the
recommended hours of service, with sample youth completing an average of 29 hours of service.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring
internal fidelity monitoring is conducted monthly for all facilitators.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIKids
(Contract Provider)
640 Roberts Road, Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment (CYT)
SPEP Review Date(s): March 7 ‐ 10, 2017
Florida Department of Juvenile Justice
Report Date(s): 11/30/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Escambia Boys Base MQI Program Code: 1271
Provider Name: AMIKids Contract Number: 10079
Location: Escambia County / Circuit: 1 Number of Beds: 28
Review Date(s): March 7 - 10, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment (CYT). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has a protocol for staff to be trained in this intervention. AMIkids Headquarters
located in Tampa, Florida, conducts Cannabis Youth Treatment (CYT) twice a year. When the
training is offered, the designated mental health clinical authority (DMHCA), sends staff to be
trained. Currently, there is one staff member trained in CYT. There is a plan to have at least one
more staff sent when the next training becomes available. The current staff member did receive
formal training in CYT in September 2016 and provided documentation to verify this information. A
second staff member was formally trained in CYT November 2015. This staff member does not
facilitate CYT groups at this time, but does assist with completing internal fidelity monitoring.
A previous staff member conducted CYT groups during the review period. This staff member was
formally trained in CYT and the program provided documentation to verify this information.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate Cannabis Youth Treatment (CYT) manual. The manual
was copied and provided to the staff during formal training. The program facilitates a combination
of the curriculum; Motivational Enhancement Therapy/Cognitive Behavioral Therapy (MET/CBT5
and CBT7). This combination is allowed per the instructor’s manual. The manual includes the
following: implementation information, facilitator requirements, session requirements, script, and
worksheets for the lessons being provided. CYT is designed for youth ages 12-18 with a
diagnosis of cannabis abuse or dependence. The youth must be experiencing emotional,
physical, legal, social, and/or academic problems associated with marijuana use.
The facility maintains a binder with with all CYT information. The first section contains the
schedule of when groups are currently held. CYT is currently held on Tuesdays from 2:00pm-
3:00pm. The second section contains all the attendance sheets. The attendance sheets included
the type of group, location of group, date, time, group topic, facilitator/credentials, youth names
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
and signatures, comment section, and facilitator signature and date. If a youth was absent, the
majority of the attendance sheets had a reason listed why the youth was absent. All of the
attendance sheets had signatures completed. The third section includes the internal fidelity
monitoring sheets for the full review period. There were internal fidelity sheets missing during the
review period. The binder also contains the internal AMI SPEP documents that are created to
send to AMI headquarters on a monthly basis. The last section of the binder had certifications for
the facilitators. The section contained education information and training certificates for CYT
facilitators. The binder was very organized and easy to find all information needed to complete the
review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program has completed three full cohorts and is currently facilitating a fourth cohort at the
time of the review. The dates of the cohorts were January 2016-April 2016, June 2016-August
2016, October 2016-December 2016. The fourth cohort was started on January 24, 2017 and
lesson nine was held on March 7, 2017. The group is running as scheduled. The majority of the
attendance sheets were accounted for. They included youth signatures, facilitator information and
if a youth was missing, the facilitator indicated why the youth missed group.
A Cannabis Youth Treatment (CYT) group was observed during the review week. Ten youth were
in attendance for this group and it was facilitated by a formally trained staff. This reviewer
obtained an attendance sheet and fidelity monitoring form for this lesson. The attendance sheet
was created by AMIkids. It included what type of therapy group, location, date of group, time,
group topic, facilitator, credentials of the facilitator, signatures of the youth, a comments section,
and the facilitator signature.
The lesson being held during the observed group was CBT9: Effective Communication. The
facilitator did a great job with the youth and effectively completed the objectives for the lesson.
The facilitator followed the lesson and was able to put the objectives and lesson into their own
words and did not read from the manual. The facilitator consistently used reflections, affirmations,
and open-ended questions. The youth were engaged, participated well, and asked appropriate
questions. The youth showed a good level of respect for this staff member. The youth and
facilitator successfully completed the role plays for this lesson. The facilitator also did a great job
debriefing the role play. The facilitator reviewed the lesson for the day and the youth were able to
discuss, in their own words, what was taught and what they learned.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Review of twelve months of documentation and interview with the designated mental health
clinical authority (DMHCA) verified that there was turnover during the review period. The previous
facilitator for Cannabis Youth Treatment (CYT) left the facility November 2016. A new therapist
was hired and trained in September 2016.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
There was a gap in service between August 2016 and October 2016, but according to
documentation, it was not due to facilitator turnover. Documentation in the CYT binder and an
interview with the DMHCA verified that the gap was due to awaiting new youth that fit the criteria
for CYT groups to enter the program. Based upon when the new staff was trained and when the
previous facilitator left, a new cohort would have been started, however due to the lack of youth to
begin a new cohort there was a gap.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There are three staff who completed internal fidelity monitoring during the review period. Two of
the three staff had been formally trained in Cannabis Youth Treatment (CYT). The DMHCA did
complete one internal fidelity monitoring, however she is not trained in CYT. This information was
verified through documentation and an interview with the DMHCA. The DMHCA is planning to be
trained in CYT to assist with internal fidelity monitoring.
One staff member was missing internal fidelity monitoring sheets for May 2016 and August 2016.
A second staff member was missing internal fidelity monitoring sheet for December 2016. Groups
were being facilitated during these time frames.
The program can earn 2 points if each facilitator is monitored every month.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The DMHCA completes weekly clinical supervision meetings with all therapists and clinical case
managers. All fidelity monitoring sheets are reviewed by the DMHCA and any corrective action
indicated on the internal fidelity form is discussed during the weekly supervision meeting.
No corrective actions were indicated on the internal fidelity monitoring forms during the review
period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The current facilitator of Cannabis Youth Treatment (CYT) has not been at the facility long enough
to have received a yearly performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 4 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 27%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 4 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (13 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 27% (4 of 15) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 18 25 72%
Totals 73 100 73%
This SPEP report evaluates Cannabis Youth Treatment (CYT), an intervention delivered at
Escambia Boys Base.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. None received at least the
recommend weeks of service, with sample youth completing an average of 8 weeks of service.
The program earned 0 points for Amount of Service: Contact Hours. None received at least the
recommended hours of service, with sample youth completing an average of 14 hours of service.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring
internal fidelity monitoring is conduct at least monthly for all facilitators.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service, possibly by adjusting the program's service schedule
to ensure that CYT is delivered in a manner that allows for the service to reach the full targeted
SPEP dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10