state: FL

summary: PREA - Year 1 Cycle 4

raw_content:
Program: Walton Academy for Growth and Change
Report Type: PREA
Fiscal Year:
Status:
PDF URL: https://www.djj.state.fl.us/content/download/654444/file/Florida_Walton Academy for Growth and Change_year1cycle4.pdf?version=1

source: DJJ

program_name_raw: Walton Academy for Growth and Change

cycle: Year 1 Cycle 4


facility_name: Walton Academy for Growth and Change

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32425
Primary Service: Aggression Replacement Training
SPEP Review Date(s): May 1- 4, 2018
Florida Department of Juvenile Justice
Report Date(s): October 5, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: WAGC MQI Program Code: 1443
Provider Name: ROP Contract Number: XXXX
Location: DeFuniak Springs County / Circuit: 1 Number of Beds: XXXX
Review Date(s): 5/1/18-5/4/18 Lead Reviewer Code: 141
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 3 # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has a training protocol of having at least three therapeutic staff trained in Aggression
Replacement Training (ART). There are currently three therapeutic staff trained to facilitate ART.
All three were formally trained in ART and the facility maintains records of the staff training. Two
therapeutic managers have consistently facilitated the group since November 2017. The clinical
director is also trained and conducts internal fidelity monitoring and can also conduct group when
needed.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Aggression Replacement Training (ART) manual for review. This
manual was the third edition-revised and expanded by authors Barry Glick and John Gibbs. The
manual was organized by each lesson: skill streaming, moral reasoning, and anger control. Each
lesson is divided by weeks. The book designates what the skill is for each lesson, lists the
objectives that should be met, materials that are needed, and then leads into the lesson. There is
a general script to follow and a procedure for each lesson to use as a guide. The skill steps are
broken into sub-steps for the facilitator to follow and steps the youth needs to complete. There are
no student manuals for ART.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Aggression Replacement Training (ART) is held Mondays and Wednesdays from 2:30pm to
3:30pm and Fridays from 3:30 to 4:30pm. ART group was observed on Wednesday, May 2, 2018
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
from 2:30pm to 3:30pm. There were nine youth in this group and this was the second group held
for this cohort. This particular cohort began on Monday, April 30th. The group started with
reviewing group expectations and norms. These were posted in the room in frames on the wall.
The youth then completed check-ins. The youth were still getting to know each other in this group.
All of the youth participated in the group activities. There was one youth who was very new to the
program that was having trouble settling into the group. The facilitator handled the youth well and
worked with him to feel more comfortable. The facilitator did a good job engaging the youth and
encouraging the youth to participate. The facilitator kept good control of the youth. The facilitator
kept to the group on task and worked well with them.
There was one staff member in the group whose cell phone rang during group. According to both,
clinical director and program director, supervisors are allowed to have cell phones on the floor of
the facility. It was suggested that if cell phones are allowed on the floor that they be on silent while
in group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There has not been facilitator turnover during the review period and there have not been any gaps
in service during this time.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process is for the clinical director to complete internal fidelity monitoring for
Aggression Replacement Training (ART). Internal fidelity monitoring forms were reviewed from
November 2017 through the current date. The months of November, January, and April were
missing for internal fidelity monitoring. ART groups have consistently been held since November,
2017.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program process for corrective actions are for any issues to be addressed during weekly
clinical supervision meetings between the clinical supervisor and facilitator. If there is a significant
issue, the facilitator is monitored during the next group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Upon review of the months of internal fidelity monitoring that were completed, there were no
significant issues noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program received a new contract in June 2017. No employees have received yearly
performance evaluations at this time. The program director provided this reviewer with a blank
copy of the yearly performance evaluation. There is a section in the yearly performance
evaluation that addresses group facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 4 youth in the
sample, 100% (4 of 4) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 4 youth in the
sample, 100% (4 of 4) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 1 youth
High = 2 youth
Total Youth in Sample = 4 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 2 youth
Total Youth in Sample = 4 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (4 of 4) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 50% (2 of 4) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 20%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 100 100 100%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 4 total youth sampled, all 4
included dosage with end dates in the EBS Module. Of those youth, 4 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 21 weeks of
service, with an average of 12 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 4 total youth
sampled, all 4 included dosage in the EBS Module. Of those youth, 4 received at least the
recommended hours of service. Youth in the sample completed between 30 and 60 hours of
service, with an average of 37 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can maintain their SPEP Quality of Service Delivery
score by continuing the processes in place at the time of this review.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

report_type: SPEP

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Date(s): May 1‐ 4, 2018
Florida Department of Juvenile Justice
Report Date(s): 05/07/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: WAGC MQI Program Code: 1443
Provider Name: ROP Contract Number: XXXX
Location: DeFuniak Springs County / Circuit: 1 Number of Beds: XXXX
Review Date(s): 5/1/18-5/4/18 Lead Reviewer Code: 141
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 7 # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's protocol is to have all of the therapeutic managers trained to facilitate Impact of
Crime (IOC). There have been four therapetuic managers trained since the implementation of the
Rite of Passage contract in June 2017. As of the week of the review, there are still three
therapeutic managers facilitating IOC. All four therapeutic managers were formally trained through
the Department of Juvenile Justice.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains four Impact of Crime (IOC) manuals that are current. There are four
sections to the IOC manual. The first section is the facilitator’s guide, which includes an
introduction and implementation section. The first section also includes how the chapters and
lessons are structured and how they should be delivered. The second section has all seven
chapters with the IOC script, lessons, and activities for the youth. The third section contains extra
worksheets, list for videos, behavior contracts, and other helpful worksheets to use during group.
The final section is the student guide.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program holds Impact of Crime groups on Mondays and Wednesdays. There have been two
completed cohorts since November 2017. A third cohort started on March 30th and was currently
running at the time of the review. The (IOC) group was not observed during the review because
the facilitators recently completed external fidelity monitoring successfully by Programming and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Technical Assistance. During the external fidelity monitoring session completed prior to this
review, the group went very well, and youth were understanding the concepts. The facilitators
were knowledgeable of the material and worked to maintain the attention of the group, despite
behavorial issues with one youth.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has had one therapeutic manager leave since June 2017 but this did not impact any
group cohort. The program has a guideline of having three therapuetic managers formally trained
to facilitate IOC.
There have been no gaps in service since November 2017 when the groups started. The clinical
director reported that the plan is to hold two groups at a time to ensure that all youth are able to
attend and complete IOC groups.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process is to have the clinical director conduct all internal fidelity monitoring.
However, the clinical director has not been trained and is not able to conduct internal fidelity. The
program currently has one of the therapeutic managers conducting internal fidelity monitoring.
Internal fidelity monitoring was not conducted for November and December. January through April
internal fidelity monitoring was completed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any facilitation issues during weekly clinical supervision meetings.
Once the issue is addressed, the facilitator is observed again during the next group. Upon review
of all of the internal fidelity monitoring logs, there were no major issues noted for the facilitators.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program received a new contract June of 2017. Yearly performance evaluations have not yet
been conducted for the facilitators for the Impact of Crime curriculum. The program director
provided a copy of the program's performance evaluations for therapeutic managers. There is a
section that addresses group facilitation abilities.
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 8 youth in the
sample, 38% (3 of 8) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 8 youth in the
sample, 0% (0 of 8) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Table 1
Moderate = 2 youth
Moderate-High = 1 youth
High = 4 youth
Total Youth in Sample = 8 youth
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 8 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (7 of 8) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 50% (4 of 8) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 8 total youth sampled, all 8
included dosage with end dates in the EBS Module. Of those youth, 3 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended weeks of service. Youth in the sample completed between 8 and 29 weeks of
service, with an average of 14 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 8 total youth sampled,
all 8 included dosage in the EBS Module. Of those youth, 0 received at least the recommended
hours of service. Youth in the sample completed between 6 and 14 hours of service, with an
average of 11 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can maintain their SPEP Quality of Service Delivery
score by continuing the practices in place at the time of this review.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 3/12/19-3/15/19
Florida Department of Juvenile Justice
Report Date(s): 7/2/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program provided training documentation for the twelve-month review period. During the last
year, one therapeutic manager and the designated mental health clinical authority facilitated
Aggression Replacement Training (ART). The facilitators were formally trained in ART in August
2017.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The group was held on Mondays, Wednesdays, and Fridays from 2:30pm-3:30pm. The youth are
chosen for this group based upon the top three criminogenic needs on the residential positive
achievement change tool. The attendance sheets are maintained in a binder labeled ART and
kept the the designated mental health clinical authority's (DMHCA) office.
The program maintains the Aggression Replacement Training manual authored by Barry Glick
and John Gibbs, third edition revised and expanded. The manual includes all lessons. Each
lesson has facilitation guidelines throughout the book.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The review team did not observe an Aggression Replacement Training due to the program not
running this group at this time. There is one trained staff, the designated mental health clinical
authority (DMHCA). Due to time contraints and other duties, the DMHCA is not able to run a
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
group at this time. This curriculum was not able to be observed; therefore, this indicator rates as
non-applicable.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has had significant turnover during the review period. There were two facilitators
during the review period and one left in September 2018. The designated mental health clinical
authority (DMHCA) completed this group. The program has not held Aggression Replacement
Training (ART) since December 2018. The DMHCA has not been able to facilitate group due to
having other job duties. The program held three cohorts of ART during the review period. The
groups averaged between five and nineyouth per group.
This reviewer met with the DMHCA and program director regarding the plans for this group. The
program will send more therapeutic managers for training once they are all hired.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the designated mental health clinical authority (DMHCA) to conduct
internal fidelity monitoring for Aggression Replacement Training (ART). The program provided
internal fidelity monitoring forms for the review period. Three of twelve forms were available to
review. No further internal fidelity monitoring forms were provided for review.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the designated mental health clinical authority (DMHCA) to address
corrective actions and facilitation issues during weekly clinical supervisions. Of the three internal
fidelity mointorings there were no issues noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided one yearly performance evaluation for review. The other facilitator left
employment before the yearly performance evaluation was completed. The yearly performance
evaluations did not contain a section for group facilitation. There was no written information on the
evaluations about facilitation of Aggression Replacement Training. This was discussed with the
program director and the designated mental health clinical authority.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 10 youth in the
sample, 100% (10 of 10) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 10 youth in the
sample, 100% (10 of 10) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 4 youth
Total Youth in Sample = 10 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 10 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (9 of 10) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 40% (4 of 10) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 88 100 88%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
40 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Duration. Of the 10 total youth sampled, all
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 10
received at least the recommended weeks of service. Youth in the sample completed between 10
and 23 weeks of service, with an average of 16 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 10 total youth
sampled,all 10 included dosage in the EBS Module. Of those youth with correct dosage, 10
received at least the recommended hours of service. Youth in the sample completed between 30
and 60 hours of service, with an average of 33 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups, by ensuring fidelity monitoring is
completed every month for each facilitator, and by ensuring each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Date(s): 3/12/19-3/15/19
Florida Department of Juvenile Justice
Report Date(s): 3/14/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's protocol is for all therapeutic managers to be formally trained to conduct Impact of
Crime (IOC) groups. The program provided training records for the six facilitators of IOC group.
Three of the therapuetic managers were formally trained September 2017, one was formally
trained June 2017, and two were formally trained September 2018. There is currently one
therapeutic manager and the designated mental health clinical authority facilitating IOC.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Impact of Crime (IOC) manual. The facilitators are using the current
manual for IOC. Each cohort was run pursuant to the training protocol. The group is currently
held on Tuesdays and Thursdays from 3:00pm-4:00pm. The attendance sheets are maintained in
a binder labeled IOC and are separated by cohort. All youth are eligible to attend IOC while at the
program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This reviewer observed Impact of Crime on Thursday, March 14, 2019. There were eight youth in
group. The group started five minutes late due to waiting on some of the youth to be brought to
group. The group ended ten minutes short. Based on the review of the attendance forms, ending
group short of the full hour was not common practice.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The group lesson was in chapter two activity two. The youth were learning about responsible
thinking, irresponsible thinking, and "stinkin thinkin". The facilitator began with a short review. For
the activity, the youth were handed five statements and they were to organize them between
responsible thinking and irresponsible thinking. Once they organized them, the youth discussed
why they made their choices. For the next activity the facilitator passed out the youth worksheets
for this lesson. The facilitator creates packets of worksheets for the youth from the youth
workbook in the manual. The youth were to come up with three of their own statements for each
type of stinkin thinkin. This took the rest of the time for group. The facilitator collected all of the
workbooks and let them know that they would review the worksheets during the next group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has experienced signficant turnover during the review period.; however, the turnover
did not affect the delivery of Impact of Crime (IOC) to youth. There had been six trained facilitators
during the review period. At the time of the annual review, there were two trained facilitators. The
facilitator observed is also leaving employment shortly after the conlusion of this review. There
were no gaps in service during the review period. The program conducted four cohorts of IOC
and is currently running a fifth cohort. There was an average of eight to ten youth in each cohort.
This reviewer met with the designated mental health clinical authority and program director to find
out the plan for IOC group once the current faciltiator leaves. The designated mental health
clinical authority requested IOC training for the current therapeutic managers to avoid any
unneccesary gaps in service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided Impact of Crime internal fidelity monitoring forms for the review period.
Three of the six facilitators received internal fidelity monitoring. Two of the therapeutic managers
were missing ten months of internal fidelity monitoring. One of the therapeutic managers had five
months of internal fidelity monitoring.
The designated mental health clinical authority (DMHCA) conducts internal fidelity monitoring.
During the review period, one of the previous therapeutic managers was designated to assist with
internal fidelity monitoring until the DMHCA was trained. This therapeutic manager is no longer at
the program. The DMHCA did not conduct internal fidelity monitoring prior to being trained.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The process for addressing corrective actions of facilitators is for the designated mental health
clinical authority (DMHCA) to address these issues during weekly individual clinical supervisions.
A review of the internal fidelity monitoring forms provided showed no corrective actions were
noted nor taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Five of the six Impact of Crime facilitators received a yearly performance evaluation during the
review period. One therapeutic manager left the program before the yearly performance
evaluation was due. The yearly performance evaluations did not contain a section for group
facilitation. There was no written information on the evaluations about facilitation of Impact of
Crime. This was discussed with the program director and the designated mental health clinical
authority.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 40 youth in the
sample, 70% (28 of 40) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 40 youth in the
sample, 60% (24 of 40) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 11 youth
High = 22 youth
Total Youth in Sample = 40 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 55%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 22 youth
Total Youth in Sample = 40 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (37 of 40) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 55% (22 of 40) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 75 85 88%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
40 youth completed the program during the review period. Of those youth, none had previously
received this intervention in another setting.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 40 total youth sampled, only
34 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 28
received at least the recommended weeks of service. Youth in the sample completed between 9
and 28 weeks of service, with an average of 22 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Contact Hours. Of the 40 total youth
sampled, only 34 included dosage in the EBS Module. Of those youth with correct dosage, 24
received at least the recommended hours of service. Youth in the sample completed between 6
and 28 hours of service, with an average of 21 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring fidelity monitoring is completed every month for each facilitator, and by
ensuring each facilitator’s performance evaluation includes discussion of their skill in delivering
this primary service.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Seeking Safety
SPEP Review Date(s): 3/12/19-3/15/19
Florida Department of Juvenile Justice
Report Date(s): 3/14/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 20%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program provided training documentation for the designated mental health clinical authority
(DMHCA) which occurred April 19, 2018. Seeking Safety is an informal training, verified by the
manual. This reviewer interviewed the DMHCA about the training conducted for the therapeutic
managers.The DMHCA conducts informal training during weekly clinical supervisions. Upon
review of the weekly clinical supervisions there was no documentation this training occurred. This
reviewer went back to the DMHCA to discuss the training protocol for Seeking Safety curriculum.
The DMHCA reported that it is not documented on the weekly clinical supervisions, but the
manual is utilized for training and has a section for group facilitation. During each group, clinical
supervision groups are discussed and there is a section on the documentation. This reviewer
suggested that when training takes place for Seeking Safety to have it documented on the group
clinical supervision log.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Seeking Safety began as a primary serivce on July 1, 2018. Seeking Safety is held as an open
group, meaning youth can join the group when they enter the program and work the curriculum at
their own pace for the duration of the program. Seeking Safety groups are held seven days a
week. Saturdays and Sundays the groups are held from 9:00am-10:00am, 10:00am-11:00am,
and 11:00am-12:00pm. Mondays and Tuesdays Seeking Safety is held from 4:00pm-5:00pm and
two groups are held at the same time. On Wednesdays Seeking Safety is held from 3:00pm-
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
4:00pm and two groups are held at the same time. On Thursdays and Fridays Seeking Safety is
held from 3:00pm-4:00pm and 4:00pm-5:00pm.
The attendance sheets are maintained in a binder labeled Rite of Passage Substance Abuse
Group Rosters. The attendance sheets are separated by each month and are in chronological
order.
The program provided the Seeking Safety manual for review. The program is using the current
Seeking Safety manual. Upon review of the manual there is a full chapter for training facilitators
and include: Treatment format, Adapting Seeking Safety to Different Contexts, Preparation,
Process, Conducting the Session, Treatment Guidelines, Problem Situations and Emergencies,
Checklist before Beginning the Treatment, and Materials for All Sessions.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This reviewer observed Seeking Safety group on Thursday, March 14, 2019 from 4:00pm-5:00pm.
There were seven youth in the group. The group was started on time and ended on time providing
a full hour for the group. The group topic was Recovery Thinking. The group began with a check-
in led by one of the youth. The story of "The Little Engine that Could" was read outloud in class.
The youth discussed how this story could help them in recovery. There were specific questions
asked of the youth to respond. The group had a good discussion and took the topic seriously. The
youth gave each other feedback after sharing.
There were no behavior issues during this group. The youth in the group were well mannered
and a had a good rapport with the facilitator of the group. Motivational Interviewing techniques
were used to allow the youth to think deeper about their recovery. Feedback, reflections, and
affirmations were also used by the facilitator and the youth. The youth take turns leading the
group each day. By leading, they conduct the group check-in, read the lesson or choose someone
to read the lesson, and conduct the check-out.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and interviews with the designated clinical mental health authority
(DMHCA) confirmed two of the previously trained therapuetic managers left during the review
period. There are currently three therapeutic managers and the DMHCA trained in Seeking Safety
curriculum. The program's protocol is to have all therapuetic managers trained in the Seeking
Safety curriculum. The program provided attendance sheets for the review period. There were no
gaps in service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program did not provide any internal fidelity monitoring for the review period for Seeking
Safety. This reviewer discussed the lack of internal fidelity monitoring for Seeking Safety with the
designated mental health clinical authority. It was reported that it wasn't done. There is not a form
for internal fidelity monitoring for Seeking Safety. It was suggested that the provider create a form
to conduct internal fidelity monitoring for Seeking Safety.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a protocol that the designated mental health clinical authority conducts weekly
clinical supervisions with all of the therapeutic managers. The clinical supervisions are conducted
in a group or individual. If there are issues with facilitation of a group it is discussed with the
facilitator during individual clinical supervision. The program did not provide internal fidelity
monitoring forms and therefore no corrective action was reviewed.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided the yearly performance evaluations for five of the facilitators. One of the
facilitators had not been at the program for a year to receive a performance evaluation.
The yearly performance evaluations did not contain a section for group facilitation. There was no
written information on the evaluations about facilitation of Seeking Safety. This was discussed
with the program director and the designated mental health clinical authority.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 25 youth in the
sample, 20% (5 of 25) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 25 youth in the
sample, 0% (0 of 25) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 7 youth
High = 14 youth
Total Youth in Sample = 25 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 56%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 25 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (22 of 25) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 56% (14 of 25) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 65 95 68%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
40 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 25 total youth sampled, all
25 included dosage with end dates in the EBS Module. Of those youth, 5 received at least the
recommended weeks of service. Youth in the sample completed between 4 and 25 weeks of
service, with an average of 12 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 25 total youth
sampled, all 25 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed 25 hours of service.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring all facilitators receive formal training prior to delivering services to youth; by
ensuring fidelity monitoring is completed every month for each facilitator; by ensuring any
corrective action needed is identified, applied, and demonstrated; and by ensuring each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Seeking Safety


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, Inc.
(Contract Provider)
285 Gene Hurley Road
DeFuniak, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 3/9/20
Florida Department of Juvenile Justice
Report Date(s): 3/9/20
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has therapeutic managers trained to conduct Aggression Replacement Training
(ART). Three therapeutic managers were trained to facilitate ART during the review period. All
documentation was viewed within the Department's learning management system, SkillPro, and in
the ART Standardized Program Evaluation Protocol (SPEP) binder. There have been two
therapeutic managers who have facilitated the ART groups over the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provides Aggression Replacement Training (ART) as a mental health group. Youth
who have a history of aggression and/or score high in aggression on the Residential Assessment
for Youth (RAY) are eligible to take the ART group.
Aggression replacement training groups are held on Mondays, Wednesdays, and Fridays from
3:00pm to 4:00pm. There is a Standardized Program Evaluation Protocol (SPEP) binder kept in
the director of student services office. The program maintains the correct manual for ART in a
therapeutic manager's office.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training (ART) was observed on Tuesday, February 4,
2020, from 3:00pm to 4:00pm.This group began January 29, 2020. The lesson is Understanding
the Feelings of Others. There are nine youth attending group; two of the nine youth are graduates
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
of the previous ART group. The facilitator invites previous graduates who demonstrated
leadership throughout the group to come and assist with facilitation of the next group. According
to an interview with the facilitator, the youth respond well and it has been helpful with participation.
The two assistant youth demonstrated the skill. All other youth, with the exception of one
demonstrated all of the skills except one. This youth was encouraged by the facilitator and the
other youth in group. If any of the youth struggled with the skill, he got support from others in the
class and worked through the skill until the concept was understood. The group ran for a full hour
and is held as a closed group per contract.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol is for the therapuetic managers to be trained in Aggression Replacement
Training (ART). There are currently three threrapuetic managers trained. A therapeutic manager
and two designated mental health clinical authorities (DMHCA) left during the twelve month
review period. The current main facilitator of ART will be leaving employment on Monday, March
9, 2020. At that time another therapeutic manager will be facilitating this group. Once a DMHCA is
hired and the therapeutic managers are hired they will need to be trained to meet the program's
protocol to have more staff trained. Per a review of the attendance sheets there have not been
gaps in service during the twelve month review period. One cohort was facilitated by a previous
therapuetic manager who left employment before the group was over; another therapeutic
manager completed the group.
There have been three cohorts completed during the review period. The first cohort was held April
15, 2019 through July 22, 2019. The group was not held in lesson order per the attendance
sheets. Three of the groups held were not held for a full hour. There were four youth who started
and ended the group. The facilitator of this group is no longer employed at the program.
The second cohort was held from August 21, 2019 to October 26, 2019. The group started and
ended with seven youth. A review of the attendance sheets confirmed these sessions were held
according to the ART manual. The third cohort was held from November 5, 2019 through January
15, 2020. Ten youth started and ended the group. There were no issues with this cohort. The
fourth cohort was started on January 29, 2020 and is continuing to meet. Ten youth started the
group; however, one was dropped due to "youth not ready for ART group". There are currently
nine youth in the group.
The issues with the first cohort were discussed with the director of student services, previous
designated mental health clinical authority, and the program director. Since then, there have not
been any issues.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There were no internal fidelity monitorings completed from March 2019 through October 2019. At
the beginning of the review period, the clinical director conducted ART groups. There were three
therapeutic managers trained as well, but did not conduct internal fidelity monitoring. The previous
designated mental health clinical authority (DMHCA) left employment in July 2019. When a
therapuetic manager completed that group, internal fidelity was not completed. When a new
DMHCA was hired, she was unsure the therapeutic managers who were trained could conduct
internal fidelity monitoring. This reviewer discussed with the clinical director only the trained
therapetuic managers can conduct internal fidelity monitoring.
Internal fidelity monitoring was conducted for the current facilitator during the month of November.
It was completed for each of the sections of ART for this facilitator. Three months or nine total
internal fidelity monitoring forms were missing for this facilitator.
In total, thirty-three internal fidelity monitorings were not completed during the review period. Of
the three completed, two of the forms did not have dates or who conducted the fidelity monitoring
and dates.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
When there are issues or a need for corrective action arises, the designated mental health
clinician authority (DMHCA) will address them during weekly clinical supervisions. The facilitator is
observed again. If the issues persists further action is taken.
The three internal fidelity monitoring forms completed had detailed notes regarding the groups
observed. There were no indications of corrective actions.
Fidelity monitoring was conducted only once during the review period. Internal fidelity monitoring
must be conducted to award points in this section.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations for the therapeutic managers include a section in the
evaluation for group facilitation.
The previous designated mental health clinician authority who was trained was not here for the
due annual evaluation due to leaving employment prior to the due date. The current facilitator has
not received a yearly performance evaluation at the time of this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 11 youth in the
sample, 64% (7 of 11) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 11 youth in the
sample, 100% (11 of 11) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 3 youth
High = 5 youth
Total Youth in Sample = 11 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 11 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (10 of 11) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (5 of 11) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 84 100 84%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
31 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 6 points for Amount of Service: Duration. Of the 11 total youth sampled, only
7 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended weeks of service. Youth in the sample completed between 14
and 20 weeks of service, with an average of 15 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 11 total youth
sampled, all 11 included dosage in the EBS Module. Of those youth, 11 received at least the
recommended hours of service. Youth in the sample completed between 30 and 60 hours of
service, with an average of 33 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups; by ensuring fidelity monitoring is
completed every month for each facilitator; and by ensuring any corrective action needed is
identified, applied, and demonstrated.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10