summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc.
(Contract Provider)
640 Roberts Drive
Bldg 3780, Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Period: 2/1/20 - 1/31/21
SPEP Review Conducted: 2/1/21
Florida Department of Juvenile Justice
Report Date: 2/26/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Aggression Replacement Training (ART) documentation was reviewed for the twelve month
review period. There are four staff trained to conduct ART groups; executive director (August
2014), clinical director (October 2012), Pensacola day treatment executive director (February
2014), and one therapist (November 2015). All training documentation was provided for this
review. There are two primary facilitators for ART groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
This program provided the Aggression Replacement Training (ART) manual for this reviewer. This
manual was the third edition-revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. The book designates what the skill is for each lesson, lists the
objectives that should be met, materials that are needed, and then leads into the lesson. There is
a general script to follow and a procedure for each lesson to use as a guide. The skill steps are
broken into sub-steps for the facilitator to follow and steps the youth needs to complete. There are
no student manuals for ART.
The program maintains the records for ART groups in a binder located in the clinical director's
office.
The program determines what youth are eligible for ART groups by the results of their Residential
Assessment for Youth results and the biopsychosocial information.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Aggression Replacement Training was not observed during this review period, therefore this
indicator rates as non-applicable. There were not enough youth to start a new cohort at the time
of this review
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover in Aggression Replacement Training (ART) facilitators during
this twelve month review period. This turnover did not effect the groups at the program. In the
event a facilitator is out on extended leave or leaves employment, a co-facilitator would continue
with the ART group to avoid disruption to the group. ART is co-facilitated on a regular basis at this
program.
Aggression Replacement Training is held on Mondays, Wednesdays, and Fridays from 1:00pm-
2:00pm. There have been three completed cohorts during this review period. All sign-in sheets
were provided for the full twelve month review period.
The first cohort was held from February 10, 2020 to April 17, 2020. There were seven youth who
started the group and six youth sucessfully completed the group. One youth was placed in
detention and did not come back to the program.
The second cohort started July 13, 2020 and ended on September 17, 2020. There were seven
youth who started the group and seven youth who ended the group.
The third cohort was held from October 12, 2020 to December 18, 2020. Six youth started the
group and six youth ended the group.
There was a gap in service between April 17, 2020 and July 13, 2020 due to a low census and not
receiving youth into the program because of COVID protocols. There is a current gap in service
starting on December 18, 2020 due to low census and the program has not received new youth.
Neither gap qualify as an unnecessary gap in services.
The program has been made aware ART groups can start with a lower than required number due
to the current COVID constraints.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The executive director of the program, executive director for the day treatment program, or clinical
director of the program conducts internal fidelity monitoring for Aggression Replacement Training
(ART) groups.
The program provided all internal fidelity monitoring forms for the full twelve month review period.
All internal fidelity monitoring was completed for the full twelve month review period. All reviewed
fidelity monitoring forms included feedback from the facilitator, co-facilitator, and the observer of
the groups.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for the internal fidelity monitoring and correction actions is for the for the
clinical director to meet with the facilitator(s) on a weekly basis. If there are any issues identified
on the internal fidelity forms it is addressed with the facilitator and reviewed during a following
group.
There were no corrective actions noted for the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Three facilitators for Aggression Replacement Training (ART) received their annual performance
evaluations. One of the three evaluations included a section to address group facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 23 youth in the
sample, 65% (15 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 23 youth in the
sample, 96% (22 of 23) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 70%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 7 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 7 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 70% (16 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 30% (7 of 23) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 15 25 60%
Totals 84 100 84%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
23 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Duration. Of the 23 total youth sampled, all
23 included dosage with end dates in the EBS Module. Of those youth, 15 received at least the
recommended weeks of service. Youth in the sample completed between 4 and 20 weeks of
service, with an average of 11 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, all 23 included dosage in the EBS Module. Of those youth, 22 received at least the
recommended hours of service. Youth in the sample completed between 10 and 60 hours of
service, with an average of 33 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 94%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Escambia Boys Base

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc.
(Contract Provider)
640 Roberts Drive
Bldg 3780, Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment
SPEP Review Period: 2/1/2020 - 1/31/2021
SPEP Review Conducted: 2/1/2021
Florida Department of Juvenile Justice
Report Date: 2/26/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program provided all training documentation for Cannabis Youth Treatment for the twelve
month review period. There have been five trained facilitators; clinical director (March 2015 and
December 2020), therapist (November 2015), two clinical case managers (December 2020).
There have been two main facilitators during the review period; the previous clinical director and
the current clinical director.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate Cannabis Youth Treatment (CYT) manual. The manual
was copied, put in a binder with a copy of the manual in the front sleeve, and provided to the staff
during formal training. The program facilitates a combination of the curriculum; Motivational
Enhancement Therapy/Cognitive Behavioral Therapy (MET/CBT5 and CBT7). This combination is
allowed per the instructor’s manual. The manual includes the following: implementation
information, facilitator requirements, session requirements, script, and worksheets for the lessons
being provided.
CYT is designed for youth ages 12-18 with a diagnosis of cannabis abuse or dependence. The
youth must be experiencing emotional, physical, legal, social, and/or academic problems
associated with marijuana use.
All documentation for groups is maintained in a binder kept in the clinical director's office.
Youth are assessed to attend CYT groups based upon the results of their Residential Assessment
for Youth indicating a need for substance abuse/use treatment or through a diagnosis of
substance use/abuse disorder.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
At the time of the annual review, Cannabis Youth Treatment was not running due to a low census
and not enough youth to begin a new cohort. A session of this service was not able to be
observed; therefore this indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There has been turnover during the twelve month review period. The previous clinical director was
the primary facilitator for Cannabis Youth Treatment. This did not cause gaps in service.
There has been a gap in service during this review period; however, it was not due to facilitator
turnover and does not qualify as an unnecessary gap. The program has experienced low
admissions during COVID. The gaps in service were between April 7, 2020 and August 4, 2020
due to a low census and youth not being admitted into the program.
All sign-in sheets were provided for review. There have been three cohorts completed during this
review period.
The first cohort was held February 4, 2020 to April 7, 2020. Eight youth began the group and
seven youth successfully completed the group. One youth was placed in detention and did not
return to the program.
The second cohort began August 4, 2020 and ended October 6, 2020. The group started with four
youth. Three youth were added to this group after the first session was completed. A review of the
program's contract showed Cannabis Youth Treatment (CYT) must be held as a closed group.
This reviewer contacted the program and AMI's national director of mental health services
regarding this information. It has been clarified with the program CYT will be held as a closed
group moving forward. Due to youth beginning and ending at different times in the previous cohort
a third cohort started with the two youth who had not completed in the August 4, 2020 to October
6, 2020 group.
A third cohort was started on September 3, 2020. Two youth successfully completed on
November 12, 2020. Two other youth successfully completed on December 15, 2020.
There is a current gap in service starting on December 15, 2020 due to no new addmissions to
the program. The program has received permission to start a group below the recommended
amount of youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for another trained facilitator or the clinical director to conduct internal
fidelity monitoring for Cannabis Youth Treatment. All internal fidelity monitoring forms were
provided for review. There was no missing fidelity monitoring for the twelve month review period.
There was feedback provided by the observer and comments made by the facilitator on the fidelity
monitoring forms.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's corrective action process is for the clinical director to address any issues or
corrective actions during weekly one-to-one meetings with the facilitators. Due to the clinical
director being the main facilitator for Cannabis Youth Treatment any issues are provided to the
executive director of the program and it is handled with the facilitator.
There were no corrective actions noted during the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The facilitator for Cannabis Youth Treatment had received their annual performance evaluation at
the time of this review. The annual evaluation form for the clinical director position does not have
a section to address group facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 12 weeks for this type of service. Of the 17 youth in the
sample, 18% (3 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 71%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 7 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 71% (12 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 41% (7 of 17) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 18 25 72%
Totals 83 100 83%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Escambia
Boys Base.
23 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Duration. Of the 17 total youth sampled, all
17 included dosage with end dates in the EBS Module. Of those youth, 3 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 20 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth, 17 received at least the
recommended hours of service. Youth in the sample completed between 15 and 29 hours of
service, with an average of 16 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 90%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Escambia Boys Base

program_status_note: Open

service_type: Cannabis Youth Treatment


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc.
(Contract Provider)
640 Roberts Drive
Bldg 3780, Corry Station
Pensacola, Florida 32511
Primary Service: Impact of Crime
SPEP Review Period: 2/1/2020 - 1/31/2021
SPEP Review Conducted: 2/1/2021
Florida Department of Juvenile Justice
Report Date: 2/26/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program provided all training documentation for the Impact of Crime facilitators. The program
has five trained staff: director of operations (August 2013), direct care (July 2019), three direct
care (November 2020).
Impact of Crime group is currently not co-facilitated. There have been three facilitators over the
twelve month review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate manual for Impact of Crime (IOC). There are four sections
to the IOC manual. The manual includes all needed worksheets and facilitator forms to conduct
IOC.
The documentation for groups is maintained in a binder kept in the clinical director's office. All
youth are eligible to attend IOC groups.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on January 14, 2021 via Microsoft Teams. There
were six youth in attendance, the main facilitator, another trained facilitator, and a direct care staff.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
This reviewer did have a copy of the lesson being covered for this group. The facilitator did a good
job conducting the session and utilizing the manual without reading verbatim. The facilitator did a
review throughout the lesson and incorporated previous lessons. The facilitator explained the
homework so the youth could understand what they were to do. The exercises were completed for
the lesson covered.
The facilitator utilized Motivational Interviewing techniques to get the youth to have an open
discussion. There were no concerns during this observation.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did not experience facilitator turnover for Impact of Crime (IOC) during this review
period. There were no gaps in service during this review period.
The program has conducted three cohorts during this review period. A fourth cohort is currently
being held. All sign-in sheets were provided for review. Impact of Crime group is to be held on
Thursdays and Sundays from 1:00pm to 2:00pm. However, the times varied on the sign-in sheets
from 1:00pm - 2:00pm, 2:00pm - 3:00pm, and 3:00pm - 4:00pm. This reviewer emailed the clinical
director about the inconsistencies with the group times. The clinical director explained the
facilitator changed shifts and another facilitator took over the group to avoid any gaps or
disruptions with the group. The clinical director also reported they are looking at how IOC is being
run and will be making some changes so it is more consistent.
The first cohort began November 25, 2019 and was completed on March 11, 2020. Twelve youth
started this group and eleven youth successfully completed the group. One youth was transferred
during this group.
The second cohort began March 16, 2020 and was completed on July 1, 2020. Four youth started
the group and three successfully completed the youth. There was no explanation on the sign-in
sheets as to why the youth was no longer in group.
The third cohort began July 30, 2020 and was completed on November 15, 2020. Six youth
started the group and three youth successfully completed the group. One youth was sent to
detention and did not return to the program. Two other youth left the program prior to completing
the group.
The fourth cohort began December 9, 2020 and is continuing at this time. Six youth started the
group and six youth remain in the group.
The program can earn 2 points if each cohort begins and ends with the same facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The current director of operations and another trained faciltiator conduct internal fidelity
monitoring for Impact of Crime groups. All fidelity forms were provided for the full twelve month
review period. There were no missing internal fidelity forms.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director reviews all of the fidelity monitoring forms for Impact of Crime groups. If there
are any issues the clinical director addresses them with the facilitator. There were no noted
corrective actions during this review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
All three facilitators had received annual performance evaluations at the time of this review. None
of the annual performance evaluations contain a section for group facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 23 youth in the
sample, 83% (19 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 23 youth in the
sample, 87% (20 of 23) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 70%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 7 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 7 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 70% (16 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 30% (7 of 23) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 15 25 60%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime, an intervention delivered at Escambia Boys Base.
23 youth completed the program during the review period. Of those youth, 23 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 23 total youth sampled, all
23 included dosage with end dates in the EBS Module. Of those youth, 19 received at least the
recommended weeks of service. Youth in the sample completed between 1 and 16 weeks of
service, with an average of 15 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, all 23 included dosage in the EBS Module. Of those youth, 20 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 1 and 37 hours of
service, with an average of 30 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 89%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring each
cohort begins and ends with the same facilitator and by ensuring each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Aggression Replacement Training
SPEP Review Period: February 1, 2020 - January 31, 2022
SPEP Review Conducted: February 18, 2022
Florida Department of Juvenile Justice
Report Date: 4/12/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Aggression Replacement Training (ART) as a primary service. The clinical
case managers and therapy staff are trained to facilitate ART groups. Eight staff were trained to
facilitate ART during the annual review period: director of treatment, two therapists, two clinical
case managers, one shift supervisor, executive director, and the compliance manager.
Seven of the staff conducted ART groups during the twelve month review period. Another trained
staff member conducts internal fidelity monitoring for ART facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Aggression Replacement Training (ART) manual for review. This
manual was the third edition-revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. The book designates what the skill is for each lesson, lists the
objectives that should be met, materials that are needed, and then leads into the lesson. There is
a general script to follow and a procedure for each lesson to use as a guide. The skill steps are
broken into sub-steps for the facilitator to follow and steps the youth needs to complete. There are
no student manuals for ART.
The program maintains the records for ART groups in a binder located in the director of
treatement's office. The binder includes sign-in sheets, group schedule, training certificates and
qualifications, and the internal fidelity monitoring for each month.
The program determines what youth are eligible for ART groups by the results of their Residential
Assessment for Youth results and the biopsychosocial information. Aggression Replacement
Training groups are held on Mondays, Wednesdays, and Fridays. A review of sign-in sheets
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
showed groups are being held at various times throughout the review period. The groups are
being held for an hour at a time.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An external fidelity observation of Aggression Replacement Training (ART) was completed on
September 22, 2021 from 2:00pm to 3:00pm. There were six youth present in the group with two
facilitators and one staff member. The staff member present is trained to conduct ART groups as
well. The group was in week five and in Anger Control Training on the date of this observation.
The lesson title: Thinking Ahead.
The facilitators worked well together to conduct the lesson. The youth were interactive throughout
the whole group. The group began with a check-in and did a review of the lesson from last week.
The youth also reviewed their hassle logs from the previous week. All youth completed their
hassle logs and shared them with the group. The facilitators completed the new skill model with
two role plays to demonstrate what the youth will be doing. The examples were appropriate for the
youth and lesson.
All youth completed their role plays in groups of two. Each youth took turns being the main actor
for their role plays. The other youth in the group had their specific skills to observe and provide
feedback for each role play. The youth did a great job with all of their tasks during this group.
The facilitators did not have to rely on their manuals, but followed along with their manuals. The
youth were engaged and interactive, asked questions when needed, and the facilitators provided
approriate feedback.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process/protocol in the event of facilitator turnover is for the co-facilitator to
continue with group facilitation and another trained staff member become the co-facilitator.
The program did experience turnover with Aggression Replacement Training (ART) facilitators
during the twelve month review period. The executive director, director of treatment, one clinical
case manager, and one therapist left during the fourth quarter review period (November 2021 -
February 2022). The clinical case manager was a primary facilitator for the ART groups. The
clinical case manager left during a cohort of ART. The group continued being facilitated by the co-
facilitator of that group. Another trained staff member assisted as a co-facilitator as well.
There was a gap in service from December 2020 to March 2021. According to the previous
director of treatment, there were not enough youth entering the program during that timeframe to
begin a group. This does not qualify as an unecessary gap in service.
Five cohots were completed during the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The first cohort started March 22, 2021 and was completed on May 27, 2021. The group began
with four youth and three youth successfully completed the group. One youth was not able to go
back into the group due to behavioral issues.
The second cohort began June 14, 2021 and was completed on August 18, 2021. Seven youth
began the group and six youth successfully completed the group. When this reviewer asked about
the one youth who did not complete, the director of treatment explained the youth did not do well
in group so the youth was able to go through the lessons alone with a facilitator checking in
periodically. This reviewer advised youth should not go through a curriculum alone without being
in a group or with a facilitator. There were no sign-in sheets or any other documentation regarding
this youth going through this curriculum.This youth did not successfully complete this curriculum.
A third cohort was started on August 23, 2021 and ended on October 29, 2021. Eight youth
started this group and seven youth successfully completed the group. One youth was working
during group time and received individual sessions after work on group days. The sign-in sheets
for the group and for the individual sessions were provided for review. One youth was arrested
during the cohort.
A fourth cohort was started on October 11, 2021 and ended on December 20, 2021. There was
one youth on the sign-in sheets. The internal Standardized Program Evaluation Protocol
completed monthly by AMI Kids headquarters had documentation one youth was placed in a
cohort due to a low-risk status with a tentative release date in December 2021. Each session for
this youth was held on various days of the week and for one hour sessions. Each of these
sessions were thirty-minutes and covered two lessons. This does not follow the fidelity of the
model. Per the sign-in sheets, the youth completed twenty-six sessions in nine weeks. One youth
does not equate to a group setting. For fidelity puposes, if a youth enters the program and will not
be able to complete a service during their time in the program, individual sessions should be
completed and it not be counted as a group.
A fifth cohort started on November 8, 2021 and ended on January 14, 2022. Five youth started
this group and six youth successfully completed the group. One facilitator conducted an individual
group for a youth who was in transition and needed to complete ART prior to leaving the program.
This youth was transferred to the program from another program. The sign-in sheets were
provided for this youth. This reviewer suggested to the program this not occur on a regular basis.
Aggression Replacement Training is held as a closed group, for fidelity purposes no youth should
be added to a cohort after the first lesson is completed.
A sixth cohort was started January 24, 2022 and continues to meet at the time of this review. The
group started with four youth and four youth continue to meet at this time.
Group times varied for all cohorts, however the cohorts were held on the scheduled days of
Monday, Wednesday, and Friday, with the exception of cohort four listed above.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program's procedure is to have another trained facilitator conduct internal fidelity monitoring
for Aggression Replacement Training (ART). There is one main observer for ART groups, the shift
supervisor.
Many of the facilitators were observed more than the required two times per month throughout the
annual review year. The program went above and beyond in conducting internal fidelity monitoring
observations. Ninety-eight internal fidelity monitoring observations were completed during the
twelve month review period and eighty-six were required.
However, based upon who was facilitating groups and when, four internal fidelity observations
were missing for one facilitator in August 2021 and September 2021. Another facilitator was
missing one observation for August 2021.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The director of treatment addresses any significant issues or concerns for the facilitators of
Aggression Replacement Training (ART). The director of treatment meets with ART facilitators on
a weekly basis.
The internal fidelity monitoring forms were provided for review and include a section for observer
feedback to the facilitator and co-facilitator. On November 17, 2021 and November 19, 2021 the
facilitators were provided low scores on the majority of the items observed. This reviewer asked
the director of treatment what happened due to the low scores. The response there was confusion
about the scoring procedures for the ART facilitators.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations include a section for group facilitation. This section does not
specify the curricula conducted by the facilitator. The narratives include: group facilitation skills,
group documentation, and knowledge of evidence based services.
Five of the facilitators received an annual performance evaluation. One facilitator received a forty-
five day evaluation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 10 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 76%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 6 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 18%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 3 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 76% (13 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 18% (3 of 17) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 10 25 40%
Totals 85 100 85%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Escambia Boys Base.
17 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Duration. Of the 17 total youth sampled, all
17 included dosage with end dates in the EBS Module. Of those youth, 17 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 20 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth, 17 received at least the
recommended hours of service. Youth in the sample completed between 30 and 60 hours of
service, with an average of 32 hours.
The program was awarded 10 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 EBB ART.pdf

service_type: Aggression Replacement Training


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Escambia Boys Base
AMIkids, Inc
(Contract Provider)
640 Roberts Drive
Bldg. 3780 Corry Station
Pensacola, Florida 32511
Primary Service: Cannabis Youth Treatment
SPEP Review Period: February 1, 2021 - January 31, 2022
SPEP Review Conducted: February 18, 2022
Florida Department of Juvenile Justice
Report Date: 4/12/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Cannabis Youth Treatment (CYT) as a primary service. The therapy staff in
the program are trained to conduct CYT groups. There were four staff trained during the annual
review period to facilitate CYT groups: director of treatment, two therapists, and one clinical case
manager. There were two main facilitators of CYT during the annual review period. The director of
treatment conducts internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the appropriate Cannabis Youth Treatment (CYT) manual. The manual
was copied, put in a binder with a copy of the manual in the front sleeve, and provided to the staff
during formal training. The program facilitates a combination of the curriculum; Motivational
Enhancement Therapy/Cognitive Behavioral Therapy (MET/CBT5 and CBT7). This combination is
allowed per the instructor’s manual. The curriculum starts with three individual sessions with a
trained therapist before creating the group. The manual includes the following: implementation
information, facilitator requirements, session requirements, script, and worksheets for the lessons
being provided.
Cannabis Youth Treatment is designed for youth ages 12-18 with a diagnosis of cannabis abuse
or dependence. The youth must be experiencing emotional, physical, legal, social, and/or
academic problems associated with marijuana use.
According to the program schedule, CYT is held on Tuesdays. CYT is held as a closed group,
meaning no youth are to be added to a cohort after the first group lesson is completed. The
documentation for CYT groups is maintained in a binder located in the director of treamtent's
office. This binder includes sign-in sheets, training documentation and qualifications, group
schedule, and internal fidelity monitoring forms.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A Cannabis Youth Treatment group has not been observed during this review period. There have
been multiple gaps in service. When this reviewer has been to the program a group was not being
held. This indicator is rated as not-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's procedure/protocol in the event of facilitator turnover is for another trained
facilitator to conduct Cannabis Youth Treatment (CYT) groups.
The program did experience turnover during this annual review period. The director of treatment
and one therapist left during the fourth quarter of the review period. Another trained therapist will
facilitate CYT groups once another group begins.
The program completed one cohort during the annual review period. This cohort started on June
8, 2021 and ended on September 21, 2021. The group started with six youth and five youth
successfully completed the group. One youth obtained employment during the and received
individual support sessions if the group was missed due to the work schedule. One youth was
arrested during the review period and did not successfully complete the curriculum.
Three other youth have received the CYT curriculum. However, it was provided individually to the
youth on the following dates: March 23, 2021 to June 28, 2021 and April 23, 2021 to June 12,
2021.
There were gaps in service during the annual review period. The first gap was from December 15,
2020 to March 23, 2021. According to the director of treatment, this gap was due to not enough
youth entering the program who qualify for the CYT curriculum. During the time from March 23,
2021 to June 8, 2021 there were not enough youth entering the program at the same time to
create a group for CYT. A second gap in service was from September 21, 2021 to the current
date. There are four youth pending a CYT group beginning.
Two youth were to start a group for CYT on September 28, 2021, as indicated in the Evidence
Based Services module within the Juvenile Justice Information System. One youth did not qualify
for the treatment, the other youth did not begin treatment. According to a review of the current
group schedule and verification from the program, this youth is pending a group starting with three
other youth at the time of this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring is conducted by the director of treatment who is formally trained in the
Cannabis Youth Treatment curriculum.
A total of eight out of eight internal fidelity monitoring observations were conducted during the
annual review period. According to the AMI internal fidelity monitoring report from October 2021,
"SPEP observation was not provided by a third trained facilitator due to there only being one
youth in the group." There was internal fidelity monitoring conducted during the months of March
to June where two cohorts were held with one youth in each.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy and procedure is to have the director of treatment address any significant
issues or corrective actions with the facilitators of Cannabis Youth Treatment (CYT). The director
of treatment meets with staff once a week.
All internal fidelity monitoring forms provided for the annual review period and there were no
indications of significant issues or corrective action provided to the facilitators.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Four trained staff received an annual performance evaluation during this twelve month review
period. On all of the evaluations there is a narrative section for group facilitation, which covers:
group facilitation skills, group documentation, and knowledge of evidence based interventions.
The specific curricula the staff are trained in is not included in the performance evaluation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 12 weeks for this type of service. Of the 12 youth in the
sample, 33% (4 of 12) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 12 youth in the
sample, 100% (12 of 12) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 12 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 25%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 3 youth
Total Youth in Sample = 12 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (11 of 12) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 25% (3 of 12) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 18 25 72%
Totals 85 100 85%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Escambia
Boys Base.
17 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Duration. Of the 12 total youth sampled, all
12 included dosage with end dates in the EBS Module. Of those youth, 4 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 17 weeks of
service, with an average of 12 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 12 total youth
sampled, all 12 included dosage in the EBS Module. Of those youth, 12 received at least the
recommended hours of service. Youth in the sample completed between 14.5 and 15 hours of
service, with an average of 15 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 92%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Escambia Boys Base can optimize their SPEP Quality of Service Delivery score by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Escambia Boys Base can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 EBB CYT.pdf