summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services
AMIkids Inc.
(Contract Provider)
4337 Saffold Road
Wimauma, Florida 33598
Primary Service: Cannabis Youth Treatment (CYT)
SPEP Review Date(s): November 1‐4, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/9/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Youth Environmental Services (YES) MQI Program Code: 388
Provider Name: AMIkids Inc. Contract Number: 10172
Location: Hillsborough County / Circuit: 13 Number of Beds: 30
Review Date(s): November 1-4, 2016 Lead Reviewer Code: 146
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment (CYT). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The review of staff training files found the program has three non-licensed clinical staff, who have
received formal training by a qualified trainer to facilitate the Cannabis Youth Treatment (CYT)
curriculum; however, only one staff member has facilitated CYT this review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Cannabis Youth Treatment curriculum and has the brand-name manual.
The manual contains lesson plans and handouts and details how the curriculum is to be
implemented.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable. The
program has delivered three groups since the last annual compliance review November 24, 2015
through January 26, 2015, March 15, 2016 through May 17, 2016, and July 5, 2016 through
September 6, 2016. The delivery of the Cannabis Youth Treatment group was not observed
during this annual review as the program has not started a new group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program had one staff deliver the Cannabis Youth Treatment curriculum since the last annual
compliance review. The groups within each cycle was facilitated by the same facilitator throughout
the cycle. There have been no gaps in service and there has not been any facilitator turnover
during this review period.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The review of the programs internal fidelity monitoring found monthly monitoring was documented
for two of the three cycles in their entirety as the program did not have any documentation of
internal fidelity monitoring for the cycle conducted March 14, 2016 and completed May 17, 2016.
The process consisted of the previous designated mental health clinician authority (DMHCA)
trained in the CYT curriculum or other CYT trained facilitator completing an electronic observation
checklist for each session they observe. The checklist includes an evaluation of the session
observed and any corrective action required based on the observation. A review of the internal
fidelity monitoring documentation provided, found no evidence any corrective action was
applicable.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a corrective action plan based on their internal fidelity monitoring; however, a
review of the internal fidelity monitoring documentation provided found no evidence corrective
action was applicable. The process covers if a facilitator is found to be non-proficient they would
receive coaching on the identified deficiency in an attempt to get them proficient. If the issue is not
resolved after coaching the facilitator would then be scheduled to attend the next scheduled
training for that specific curriculum. If the facilitator is found to still be non-proficient, they would be
placed on a corporate corrective action plan for job performance. In addition, the facilitator would
be suspended from delivering the identified curriculum. Further non-proficiency may result in
disciplinary action up to and including termination of employment.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program conducts ninety-day and annual evaluations on each trained facilitators’ ability to
deliver the Cannabis Youth Treatment (CYT) curriculum. One of the trained facilitators just
recently completed the required training and was not due for an evaluation on their delivery of the
CYT curriculum as they have not delivered or co-facilitated the service. The evaluations of the
remaining two trained facilitators found one of two included an assessment of the facilitator’s skill
in delivering the CYT curriculum and the second facilitator’s evaluation did not identify a specific
curriculum.
The program can earn 1 point by ensuring that all facilitators' performance evaluations include an
assessment of their delivery of the specific intervention.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 5 youth
High = 8 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (14 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 53% (8 of 15) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 68 100 68%
This SPEP report evaluates Cannabis Youth Treatment (CYT), an intervention delivered at Youth
Environmental Services (YES).
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring that internal fidelity monitoring is conducted at least monthly for all facilitators and by
ensuring that all facilitators' performance evaluations include an assessment of their delivery of
the specific intervention.
The program earned 0 points for Amount of Service: Duration. Of the 15 total youth sampled, only
11 included dosage with end dates in the EBS Module. Of those youth with correct dosage, none
received at least the recommended weeks of service. Youth in the sample completed between10
and 11 weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 13 and 14 hours of service.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Youth Environmental Services (YES) can optimize their SPEP Quality of Service Delivery score
by ensuring that internal fidelity monitoring is conducted at least monthly for all facilitators and by
ensuring that all facilitators' performance evaluations include an assessment of their delivery of
the specific intervention.
Youth Environmental Services (YES) can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

facility_name: youth environmental services

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services (YES)
AMIkids Inc.
(Contract Provider)
4337 Safford Rd.
Wimauma, Florida 33598
Primary Service: Aggression Replacement Training
SPEP Review Date(s): December 5 ‐ 8, 2017
Florida Department of Juvenile Justice
Report Date(s): December 11, 2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Youth Environmental Services Program MQI Program Code: 388
Provider Name: AMIkids Inc., Contract Number: 10172
Location: Hillsborough County / Circuit: 13 Number of Beds: 32
Review Date(s): December 5-8, 2017 Lead Reviewer Code: 140
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers 5 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 10 # Personnel Records
Fidelity Monitoring Documents Program Schedules 10 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program currently has four facilitators trained in the Aggression Replacement Training (ART)
curriculum. The program had a fifth facilitator who provided ART groups to the youth in the
program over the last year; however, the facilitator is no longer employed by the program. The
other four facilitators at the program are the therapist, the director of treatment services, a case
manager, and the program compliance specialist. A review of the facilitators training records
showed each facilitator was trained in the ART curriculum. A qualified trainer trained all facilitators
in the primary service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
All youth admitted to the program who are identified as having a lack of pro-social skills,
aggression, or poor decision making skills after the completion of the comprehensive assessment
are placed into the Aggression Replacement Training (ART) groups. Also, youth whose therapist
believe they need the curriculum based on treatment planning or parent/guardian feedback are
placed into an ART group. The program has a written manual detailing the primary service entitled
Aggression Replacement Training – A Comprehensive Intervention for Aggressive Youth, third
edition. The ART manual is detailed and explains how each specific session is to be delivered,
which includes an outline for each group session. Each facilitator maintains a copy of the group
sign-in sheets in a binder entitled ART and has their own ART manual.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The primary service was being held every Wednesday at 2:30 p.m. Prior to observation of the
group, the facilitator provided a copy of the lesson plan from the manual to the observer. The
lesson being covered was the last session of ART, which covered moral reasoning. There was
one facilitator, one direct care staff member, and five youth in the group. The facilitator did not
read the lesson verbatim but used their own words to convey the meaning of the topic and did so
in an easy to follow manner. The group sat in a semi-circle around the facilitator so each person
could see each other. The facilitator engaged the youth to participate in the lesson and the youth
actively participated in the group by giving examples of moral reasoning. The facilitator positively
reinforced the youth’s answers to the handouts, and was very encouraging to the youth.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has a policy which addresses facilitator turnover. The policy stipulates if a facilitator
is leaving the program, leaves suddenly, or will be on leave, they explain to the youth why another
facilitator has taken over. A review of documentation indicated all groups had a main facilitator
and then a co-facilitator for each cycle of ART taught. The documentation indicates a co-facilitator
of one cycle of ART left the program during the month of August 2017. An interview with the
director of treatment services indicated the youth were informed the co-facilitator was no longer
employed by the program and a new co-facilitator was taking over. There have been no gaps in
service delivery and no turnover of facilitators since the last annual review, which was disruptive
to the services provided to the youth.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The director of treatment services, case manager, or the program’s compliance specialist
performed all fidelity monitoring for the ART groups. All fidelity monitors were trained in ART. The
program provided twelve months of internal fidelity monitoring reports to confirm they have been
conducting fidelity monitoring, for the four facilitators. The program uses an internal fidelity
checklist to document fidelity monitoring. All completed fidelity checklists are maintained in a
binder entitled fidelity monitoring for ART, which is organized by monthly tabs. There were thirty-
two instances of facilitation by four different facilitators, which required monthly fidelity monitoring.
The program had documentation of twenty-six instances of fidelity monitoring conducted by a
trained monitor; therefore, there were six instances where a facilitator taught ART during a month,
where they did not receive fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. The program uses internal fidelity monitoring
corrective action checklists to specify if a facilitator needs corrective action or improvement. A
review of the completed checklists for the last twelve months indicated there was no corrective
action or improvement noted for the facilitators who conducted ART groups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the five facilitators who conducted ART groups over the last
year found each of the employees received an evaluation. Four of the evaluations were annual
evaluations and one was a ninety-day evaluation. All evaluations contained an entry on the
evaluation for the facilitators’ abilities to demonstrate the appropriate level of skills when delivering
the primary group curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 10 weeks for this type of service. Of the 13 youth in the
sample, 54% (7 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 13 youth in the
sample, 100% (13 of 13) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 4 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 31%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 4 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (12 of 13) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 31% (4 of 13) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 89 100 89%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Youth
Environmental Services.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Insitute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 Services (15 weeks; 45
hours).
The program earned 4 points for Amount of Service: Duration. Of the 13 total youth sampled, all
13 included dosage with end dates in the EBS Module. Of those youth, 7 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 11 weeks of
service, with an average of 10 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 13 received at least the
recommended hours of service. Youth in the sample completed an average of 30 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Youth Environmental Services can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Youth Environmental Services can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Youth Environmental Services

program_status_note: Closed

service_type: Aggression Replacement Training


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services (YES)
AMIkids Inc.
(Contract Provider)
4337 Safford Rd.
Wimauma, Florida 33598
Primary Service: Cannabis Youth Treatment
SPEP Review Date(s): December 5 ‐ 8, 2017
Florida Department of Juvenile Justice
Report Date(s): December 11, 2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Youth Environmental Services Program MQI Program Code: 388
Provider Name: AMIkids Inc., Contract Number: 10172
Location: Hillsborough County / Circuit: 13 Number of Beds: 32
Review Date(s): December 5-8, 2017 Lead Reviewer Code: 140
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers 5 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 10 # Personnel Records
Fidelity Monitoring Documents Program Schedules 10 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment. The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has four facilitators trained in the Cannabis Youth Treatment (CYT) curriculum. Two
of the four facilitators are therapists at the program. The remaining two facilitators are the director
of treatment services, and the program's compliance specialist. Two of the four facilitators
conducted groups during the last twelve month period. A review of the facilitators training records
showed each facilitator was trained in the CYT curriculum. A qualified trainer trained all facilitators
in the primary service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
All youth admitted to the program who are identified as in need of substance abuse treatment
services are placed into Cannabis Youth Treatment (CYT) groups. The program has a written
manual detailing the primary service and the program is using the U.S. Department of Health and
Human Services guide for Cannabis Youth Treatment, volumes one and two. The first two
sessions of the curriculum are individual treatment session with the facilitator, and then the youth
start group sessions. The CYT manual is detailed and explains how each specific session is to be
delivered, which includes an outline for each group session. Each facilitator maintains a copy of
the group sign-in sheets and has their own CYT manual.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
CYT groups were being held every Tuesday at 2:30 p.m. Prior to observation of the group, the
facilitator provided a copy of the lesson plan from the manual to the observer. The lesson plan
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
outlined the lesson being covered, which allowed the observer to ensure the facilitator was
adhering to the model. The lesson being covered was session eleven: depression management.
There was one facilitator, one direct care staff member, and five youth in the group. The facilitator
did not read the lesson verbatim but used their own words to convey the meaning of the topic and
did so in an easy to follow manner. The group sat in chairs in a circle so each person could see
each other. The facilitator engaged the youth to participate in the lesson and the youth actively
participated in the group by giving examples of depression and coping skills.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation confirmed there was no staff turnover in the three full
cycles of Cannabis Youth Treatment, and in the current cycle being taught. The program does
have a policy for facilitator turnover. The policy stipulates if a facilitator is going to be leaving the
program or will be on leave, they explain to the youth why another facilitator will be taking over.
There have been no gaps in service delivery and no turnover of facilitators since the last annual
review.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The director of treatment services performed all fidelity monitoring for the CYT curriculum except
for one month when the compliance specialist conducted the fidelity monitoring. Both fidelity
monitors have received certification, by a qualified trainer, in CYT and fidelity monitoring. The
program provided twelve months of internal fidelity monitoring records for the two facilitors. The
program uses an internal fidelity checklist to document fidelity monitoring. The program had
fidelity checklists completed for all facilitators every month they completed CYT groups. All
completed fidelity checklists are maintained in a binder entitled fidelity monitoring for CYT, which
is organized by monthly tabs. Each monthly tab had a copy of the fidelity checklist completed on
the facilitator or facilitators who performed groups during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. The program uses internal fidelity monitoring
corrective action checklists to specify if a facilitator needs corrective action or improvement. A
review of the completed checklists for the last twelve months indicated there was no corrective
action or improvement noted for the facilitators who conducted CYT groups.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the two facilitators and the two staff who conducted fidelity
monitoring found all four received an evaluation within the last year. Three of the evaluations were
annual evaluations and one was a ninety day evaluation. All four evaluations contained an entry
on the evaluation for the facilitators’ abilities to demonstrate appropriate level of skills when
delivering the group curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 7 youth in the
sample, 0% (0 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 7 youth in the
sample, 0% (0 of 7) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 3 youth
High = 2 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 29%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 2 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (6 of 7) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 29% (2 of 7) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 18 25 72%
Totals 73 100 73%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Youth
Environmental Services.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 7 total youth sampled, all 7
included dosage with end dates in the EBS Module. Of those youth, 0 received at least the
recommended weeks of service. Youth in the sample completed 11 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 7 total youth sampled,
all 7 included dosage in the EBS Module. Of those youth, 0 received at least the recommended
hours of service. Youth in the sample completed 14 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Youth Environmental Services can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Youth Environmental Services can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Cannabis Youth Treatment


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services
AMI Kids, Inc.
(Contract Provider)
4337 Saffold Road
Wimauma, Florida 33598
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 09/11/18-09/14/18
Florida Department of Juvenile Justice
Report Date(s): 09/12/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
At the time of the annual compliance review, the program had four staff members trained to
facilitate the Aggression Replacement Training (ART) curriculum. Two staff facilitated the group
during the review period - one Therapist, and one Compliance Specialist. Both the Director of
Treatment Services and the Compliance Specialist conducted fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is uses ART-A Comprehensive Intervention for Aggressive Youth, 3rd Edition
curriculum. The ART manual is detailed and explains how each specific session is to be delivered
and includes an outline for each group session. The faciliators maintain a copy of their group's
sign in sheets and have their own manual.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Aggression Replacement Training group was observed during the review on 09/12/18. The
facilitator provided a copy of the lesson for the observer from the manual, did not read the lesson
verbatim, used her own words to convey the meaning of the discussed topic, and did so in an
easy to follow manner. The facilitator adhered to the lesson in every aspect.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Treatment
Services confirmed there was no staff turnover during the review period that adversly affected
services of Aggression Replacement Training to the youth.The program, in the case of the
turnover, utilizes administrative or other trained staff to facilitate the curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The Director of Treatment Services as well as the Compliance Specialist performed fidelity
monitoring. Both of the fidelity monitors have received certification by a qualified trainer in the
Aggression Replacement Training (ART) curriculum and fidelity monitoring. The program provided
internal fidelity monitoring records for the review period for the facilitators. The program uses an
internal fidelity checklist specific to ART to document the monitoring. The program had completed
checklists for all faciliators for each month they completed their ART groups. All completed fidelity
checklists are maintained in a binder entitled "Fidelity Monitoring for ART", which is organized by
monthly tabs. Each tab had a copy of the fidelity checklist completed on the facilitator or
facilitators who performed groups during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. An interview with the Director of Treatment
Services and a review of documentation verified the program has a process for corrective actions
based on the fidelity monitoring conducted. The program uses internal fidelity monitoring /
corrective action checklists specific to Aggression Replacement Training. The program did not
have any corrective action noted nor taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the two facilitators and the one staff who conducted fidelity
monitoring found that all three received an evaluation within the last year.The performance
evaluation of the trained facilitators found entries on the evaluations of the staff's performance in
delivering Aggression Replacement Training. These included comments on any strengths or
weaknesses of which the facilitator delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 16 youth
High = 8 youth
Total Youth in Sample = 30 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 27%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 8 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (27 of 30) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 27% (8 of 30) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 18 25 72%
Totals 93 100 93%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Youth
Environmental Services.
52 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 10 points for Amount of Service: Duration. Of the 30 total youth sampled, all
30 included dosage with end dates in the EBS Module. Of those youth, all 30 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 18 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 30 total youth
sampled, all 30 included dosage in the EBS Module. Of those youth with correct dosage, all 30
received at least the recommended hours of service. Youth in the sample completed between 30
and 45 hours of service, with an average of 31 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Youth Environmental Services can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Youth Environmental Services can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services
AMI Kids, Inc.
(Contract Provider)
4337 Saffold Road
Wimauma, Florida 33598
Primary Service: Cannabis Youth Treatment
SPEP Review Date(s): 09/11/18-09/14/18
Florida Department of Juvenile Justice
Report Date(s): 09/12/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment. The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
At the time of the annual compliance review, the program had five staff members trained to
facilitate the Cannabis Youth Treatment (CYT) curriculum. Two staff facilitated the group during
the review period - one Therapist, and one Compliance Specialist. The Director of Treatment
Services conducted the fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the Cannabis Youth Treatment (CYT) curriculum from the U.S. Department
of Health and Human Services, volumes one and two. The first two sessions of the curriculum are
individual treatment sessions with the facilitator, and then the youth begin group sessions. The
CYT manual is detailed and explains how each specific session is to be delivered, which includes
an outline for each group session. The facilitators maintain a copy of their group's sign in sheets
and have their own CYT manual.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Cannabis Youth Treatment curriculum was unable to be observed during the review dates as
the group was not running. Therefore this indicator rates as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation provided and an interview with the Director of Treatment
Services confirmed there was no staff turnover during the review period that adversely affected
services of Cannabis Youth Treatment to the youth. The program, in the case of turnover, utilizes
administrative or other trained staff to facilitate the curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The Director of Treatment Services as well as the Compliance Specialist performed fidelity
monitoring. Both of the fidelity monitors have received certification by a qualified trainer in the
Cannabis Youth Treatment curriculum and fidelity monitoring. The program provided internal
fidelity monitoring records for the review period for the facilitators. The program uses an internal
fidelity checklist specific to Cannabis Youth Treatment to document the monitoring. The program
had completed checklists for all facilitators for each month they completed their CYT groups. All
completed fidelity checklists are maintained in a binder entitled "Fidelity Monitoring for CYT",
which is organized by monthly tabs. Each tab had a copy of the fidelity checklist completed on the
facilitator or facilitators who performed groups during the specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. An interview with the Director of Treatment
Services and a review of documentation verified the program has a process for corrective actions
based on the fidelity monitoring conducted. The program uses internal fidelity monitoring /
corrective action checklists specific to Cannabis Youth Treatment. The program did not have any
corrective action noted nor taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the two facilitators and the one staff who conducted fidelity
monitoring found that all three received an evaluation within the last year. The performance
evaluation of the trained facilitators found entries on the evaluations of the staff's performance in
delivering Cannabis Youth Treatment. These included comments on any strengths or weaknesses
of which the facilitator delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 12 weeks for this type of service. Of the 25 youth in the
sample, 72% (18 of 25) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 25 youth in the
sample, 100% (25 of 25) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 9 youth
High = 10 youth
Total Youth in Sample = 25 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (18 of 25) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 40% (10 of 25) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 94 100 94%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Youth
Environmental Services.
52 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 6 points for Amount of Service: Duration. Of the 25 total youth sampled, all
25 included dosage with end dates in the EBS Module. Of those youth, 18 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 13 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 25 total youth
sampled, all 25 included dosage in the EBS Module. Of those youth, all 25 received at least the
recommended hours of service. Youth in the sample completed 14.5 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Youth Environmental Services can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Youth Environmental Services can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Youth Environmental Services
AMIkids, Inc
(Contract Provider)
4337 Saffold Road
Wimauma, Florida 33598
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 10/23/19
Florida Department of Juvenile Justice
Report Date(s): 10/28/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
At the time of the annual compliance review, the program had six staff members trained to
facilitate the Aggression Replacement Training curriculum - one Director of Treatment Services,
one Regional Compliance Director and four Therapists. Three staff actually facilitated the group
during the review period. Each staff was trained by a qualified trainer prior to delivering services
to youth.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses Aggression Replacement Training (ART) - A Comprehensive Intervention for
Aggressive Youth, 3rd Edition curriculum. The ART manual is detailed and explains how each
specific session is to be delivered and includes an outline for each group session. The faciliators
maintain a copy of their group's sign in sheets and have their own manual.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Aggression Replacement Training group was observed during the review on Wednesday,
10/23/19 from 2:30 pm to 3:30 pm. The facilitator provided a copy of the lesson for the observer
from the manual, did not read the lesson verbatim, used her own words to convey the meaning of
the discussed topic, and did so in an easy to follow manner. The facilitator adhered to the lesson
in every aspect.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the program's documentation and an interview with the Director of Treatment Services
confirmed there was some staff turnover during the review period; however, the turnover did not
adversely affect the services of Aggression Replacement Training to the youth. The program, in
the event of turnover, utilizes administrative or other trained staff to facilitate the curriculum. There
were two gaps in services due to not having enough new youth enrolled to start new groups. The
gaps were from 10/20/18-11/25/18, 38 days and from 05/09/19-06/24/19-46 days. A review of the
Juvenile Justice Information System confirmed an average daily population of 29 youth.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Staff who are formally trained in the Aggression Replacement Training (ART) currciulum
conducted fidelity monitoring and have received certification by a qualified trainer in the
curriculum. The program provided internal fidelity monitoring records for the review period for the
facilitators. The program uses an internal fidelity checklist specific to ART to document the
monitoring. The program had completed checklists for all faciliators for each month they
completed their ART groups. All completed fidelity checklists are maintained in a binder entitled
"Fidelity Monitoring for ART", which is organized by monthly tabs. Each tab had a copy of the
fidelity checklist completed on the facilitator or facilitators who performed groups during the
specific month.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. An interview with the Director of Treatment
Services and a review of documentation verified the program has a process for corrective actions
based on the fidelity monitoring conducted. The program uses internal fidelity monitoring /
corrective action checklists specific to Aggression Replacement Training. The program did not
have any corrective actions noted nor taken during the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of employee records for the two facilitators and the one staff who conducted fidelity
monitoring found all three received an evaluation within the last year. The performance
evaluations of the trained facilitators found entries on the evaluations of the staff's performance in
delivering Aggression Replacement Training. These included comments on any strengths or
weaknesses of which the facilitator delivering the curriculum should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 25 youth in the
sample, 100% (25 of 25) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 25 youth in the
sample, 100% (25 of 25) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 6 youth
High = 10 youth
Total Youth in Sample = 25 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 25 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (20 of 25) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 40% (10 of 25) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 95 100 95%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Youth
Environmental Services.
35 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Duration. Of the 25 total youth sampled, all
25 included dosage with end dates in the EBS Module. Of those youth, all 25 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 12 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 25 total youth
sampled, all 25 included dosage in the EBS Module. Of those youth, all 25 received at least the
recommended hours of service. Youth in the sample completed 30 hours of service.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Youth Environmental Services can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Youth Environmental Services can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9