summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Seeking Safety
SPEP Review Period: March 15, 2021 - March 14, 2022
SPEP Review Conducted: March 28, 2022
Florida Department of Juvenile Justice
Report Date: 7/7/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 20%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seeking Safety as a substance abuse treatment for youth in the substance
abuse part of the program. The program's training protocol is for therapeutic staff to be trained in
the Seeking Safety curriculum. The program's regional clinical director is able to train the
therapeutic staff. All trained staff have conducted Seeking Safety groups during the annual review
period.There have been six trained staff during the annual review period: five therapists and one
clinical director.
The clinical director conducts internal fidelity monitoring observations for the Seeking Safety
groups. The clinical director left employment during the annual review period. The regional clinical
director has been conducting internal fidelity monitoring observations since that time.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for Seeking Safety. Chapter two of the manual
provides all instruction information for facilitating the curriculum. All lessons and worksheets are
located in the manual. The manual is kept in the clinical director's office.
The schedule for Seeking Safety has changed during the annual review period. From March 2021
to December 2021, Seeking Safety was held on Mondays and Fridays from 3:00pm to 4:00pm
and 4:00pm to 5:00pm. From December 2021 to the current time Seeking Safety groups are held
on Fridays from 3:00pm to 4:00pm and from 4:00pm to 5:00pm. Groups on Saturdays are from
10:00am to 11:00am and 1:00pm to 2:00pm. A review of sign-in sheets confirmed groups are
being held as scheduled on a consistent basis.
The documentation for the Seeking Safety groups are maintained in a binder in the clinical
director's office. The binder includes training documentation, sign-in sheets, and internal fidelity
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
monitoring forms. Youth with a substance abuse diagnosis are eligible to attend the Seeking
Safety groups. Seeking Safety are held as open groups, meaning the youth can begin in a group
upon admission to the program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on Friday, April 1, 2022 from 3:00pm to 4:00pm. The
lesson for the group was Respecting Your Time. Group was held from 3:15pm to 4:20pm. Group
started late due to the facilitator having two intakes on the day of group and was the only therapist
present at the time the youth were admitted to the program.
The group, as a whole, reviewed the group norms. The faciltiator introducted the topic of group.
The facilitator struggled to provide some examples to connect to the topic. There were no
significant behavioral issues. One youth was obstinant to begin with, but became more focused as
group went on and participated well throughout the group. Three of the other youth showed
leadership skills and attempted to redirect this youth back to group. The group had a good
discussion on ways they could utilize their time better once out of the program.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol, in the event of facilitator turnover, is for another trained staff to continue
conducting the Seeking Safety groups. The program experienced turnover; however, the turnover
did not adversely affect services to youth. The clinical director and two therapeutic managers left
employment during the twelve-month review period. One of the therapists returned to work in
January 2022 and is still trained in the Seeking Safety curriculum.
Seeking Safety is an open group, therefore the staff did leave during a cohort. The therapeutic
managers rotate group faciltiation so the youth are familiar with all of the facilitators. All sign-in
sheets were provided for review and there were no gaps in service for the Seeking Safety groups.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the clinical director to conduct internal fidelity monitoring
observations for the Seeking Safety facilitators. The clinical director left employment during the
annual review period. Since that time, the regional clinical director has been conducting internal
fidelity monitoring for the Seeking Safety groups.
Twenty-three out of twenty-five internal fidelity monitoring observations were conducted during the
annual review period. Two months were missing for one facilitator in April and May of 2021.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process is for the clinical director to address any significant issues or corrective
actions with the facilitators of the Seeking Safety curriculum. The clinical director left employment
during the annual review period. The regional clinical director has been at the program to assist
with any facilitator issues.
The internal fidelity monitoring forms include an section to document any significant issues or
corrective actions. A review of the internal fidelity monitoring forms revealed no significant issues
or corrective actions were noted for any of the facilitators during the annual review period.
One of the fidelity forms noted on January 28, 2022, three youth in the group should repeat
Seeking Safety due to behaviors. The behaviors of these youth were documented on the internal
fidelity monitoring form for that date.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluation includes a section to document group facilitation.
Three of the six facilitators had received annual performance evaluations at the time of this
review. Three of the trained staff left employment and did not receive an annual performance
evaluation. The three annual performance evaluations listed the curriculum the employees are
trained in and included an evaluation of their group facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 90% (18 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 65% (13 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 8 youth
High = 6 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 6 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (16 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 30% (6 of 20) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 17 25 68%
Totals 81 95 85%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
31 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 20 total youth sampled, all
20 included dosage with end dates in the EBS Module. Of those youth, 18 received at least the
recommended weeks of service. Youth in the sample completed between 7 and 41 weeks of
service, with an average of 32 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, all 20 included dosage in the EBS Module. Of those youth, 13 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 31 and 87 hours of
service, with an average of 49 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 94%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring fidelity monitoring is completed every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 WAGC SS.pdf


state: FL

facility_name: Walton Academy for Growth and Change

summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Period: March 15, 2021 - March 14, 2022
SPEP Review Conducted: March 28, 2022
Florida Department of Juvenile Justice
Report Date: 7/7/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Aggression Replacement Training (ART) as a primary service. The program
trains therapists to facilitate the ART groups on a regular basis. Rite of Passage has a qualified
internal trainer who trains the program staff.
During the twelve-month review period, five staff were trained to facilitate ART groups: one
regional clinical director, one clinical director, and three therapeutic managers. Two of the staff
facilitated ART groups during the annual review period. All training records are maintained in the
SPEP binder in the clinical director's office and were provided for review.
The regional clinical director and previous clinical director conducted internal fidelity monitoring
during the annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for the Aggression Replacement Training (ART)
curriculum. The program maintains the ART manual authored by Barry Glick and John Gibbs,
third edition, revised and expanded. The manual includes all lessons. Each lesson has facilitation
guidelines throughout the book. There are no separate manuals for the youth in ART groups.
Aggression Replacement Training is scheduled for Tuesdays, Wednesdays, and Thursdays from
3:00pm to 4:00pm. All records are maintained in a binder located in the clinical director's office.
The binder includes: training records, sign-in sheets, and internal fidelity monitoring forms. These
are separated into cohorts with the youth's names, dates of the cohort, and the facilitator of the
group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The youth who receive the ART curriculum are those who have aggression issues indicated on
their Residential Assessment for Youth (RAY) results. Aggression Replacement Training groups
are closed groups, meaning youth cannot be added to a group once the first lesson has been
completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Aggression Replacement Training (ART) group was conducted on Thursday,
July 29, 2021 from 3:00pm to 4:00pm.
There was one staff, one facilitator and seven youth present for this group. The group was
completing the Moral Reasoning lesson of Sam's Problem Situation. The group rules were
reviewed and a check-in was conducted. The facilitator was prepared for group by having the
reading and worksheets ready for the group and the chart for todays group was made ahead of
time. The youth took turns volunteering to read. After each paragraph the facilitator asked
questions to ensure the youth understood the point of the paragraph(s). All youth participated in
the group and in the discussions.
The facilitator appeared comfortable conducting group. She had a copy of the lesson to follow and
was able to put the lesson into her own words. The facilitator conducted the lesson with fidelity.
The facilitator kept good control of the class and brought the kids back to the lesson when they
would have side conversations. The facilitator encourages the youth and utilizes reflections during
group discussion.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of turnover for group facilitators is for another facilitator to
continue with the Aggression Replacement Training (ART) group. Best practice is for the group to
be co-facilitated or for the facilitator to leave employment, if possible, after the group has been
completed.
The program experienced facilitator turnover; however, it did not adversely affect the groups. The
clinical director and one therapeutic manager left employment during the twelve month annual
review period. The therapeutic manager returned to employment January 13, 2022 and is able to
continue conducting ART groups. The staff left between cohorts and therefore it did not affect the
groups being held at the time of their departure.
There were two gaps in service. One was from December 10, 2020 to March 16, 2021 and one
from August 26, 2021 to September 15, 2021. A discussion with the regional clinical director
suggested the gaps in service were due to a low census of youth during those times.
The program completed five cohorts during the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The first cohort started March 16, 2021 and ended on May 20, 2021. The group began and ended
with three youth.
The second cohort started June 22, 2021 and ended on August 26, 2021. The groups began with
seven youth and six youth successfully completed the group. One youth was released prior to
completion and did not successfully complete ART group.
The third cohort started on June 22, 2021 and ended on August 26, 2021. The group began and
ended with seven youth.
The fourth cohort started on September 28, 2021 and ended on December 9, 2021. The group
began with eight youth and six youth successfully completed the group. One youth stopped
attending the group after November 10, 2021 with no explanation. The second youth was dropped
from the group due to behavioral issues and refusing to attend group. This youth will be added to
a new cohort and will have to complete the group prior to release, per the regional clinical director.
The fifth cohort started on December 20, 2021 and ended on March 3, 2022. Ten youth started
the group and seven youth successfully completed the group. This group was in quarantine from
January 11, 2022 to January 14, 2022. The group resumed on January 18, 2022. Three youth
were arrested while attending the ART group. These youth did not return to the program. One
youth refused five sessions of ART and had one removal from group. This reviewer spoke with
the regional clinical director. The youth was allowed to continue in the group and received
individual sessions for the sessions refused. Those individual session were documented on
separate sign-in sheets and provided for review. All dates correspond with the missed sessions,
with the exception of no sign-in sheet for January 4, 2022.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for the clinical director to conduct internal fidelity observations for the
Aggression Replacement Training (ART) groups. The program's Delinquency Interventions Table
in the contract states the clinical director will conduct four internal fidelity monitoring observations
each month for each facilitator. The developers of the curriculum recommend three observations.
The clinical director left employment during the twelve month review period. The regional clinical
director has been conducting internal fidelity monitoring observations for the ART groups.
There were two facilitators during the twelve month review period. There were five missing
internal fidelity monitoring observations for the twelve month review period; one in March 2021,
two for June 2021, and two for September 2021. There were seven months out of twelve where
the facilitators had additional fidelity observations.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring. Prior to leaving employment, the clinical director addressed any
signficant issues or corrective actions for the facilitators of Aggression Replacement Training
(ART). Since the clinical director left employment, the regional clinical director addresses any
significant issues or corrective actions for the facilitators of ART.
The internal fidelity monitoring forms include a section to document significant issues or corrective
actions. A review of the fidelity forms found the facilitators did receive feedback on all forms.
There are still some issues with the facilitation of the group by one individual. The regional clinical
director is monitoring this facilitator more often than required and continuing to work with this
facilitator.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
One of the two facilitators received an annual performance evaluation. One facilitator left
employment and returned a few months later. This facilitator has not been back long enough to
have received an annual performance evaluation.
The other facilitator did receive an annual performance evaluation. The evaluation form includes a
list of all curricula the employee is trained in and facilitates. The facilitator was evaluated for group
facilitation of Aggression Replacement Training during the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 14 youth in the
sample, 93% (13 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 14 youth in the
sample, 93% (13 of 14) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 7 youth
High = 5 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 36%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (12 of 14) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 36% (5 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 94 100 94%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
31 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Duration. Of the 14 total youth sampled, all
14 included dosage with end dates in the EBS Module. Of those youth, 13 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 11 weeks of
service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 13 received at least the
recommended hours of service. Youth in the sample completed between 27 and 30 hours of
service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring fidelity monitoring is completed every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

file_name: FY2021-22 WAGC ART.pdf

program_name_raw: Walton Academy for Growth and Change

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Period: March 15, 2021 - March 14, 2022
SPEP Review Conducted: March 28, 2022
Florida Department of Juvenile Justice
Report Date: 7/7/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. The program's protocol is for the
coach counselors to be trained to facilitate IOC groups. Ten trained staff facilitated IOC groups
during the twelve month annual review period: one therapeutic manager, two shift supervisors,
one regional compliance director, one director of student services, one clinical director, one
transition specialist, and three case managers. One of the ten staff was from another program
who came to Walton Academy for Growth and Change to assist during facilitator turnover. All
training documentation was provided for review.
The facilitators observe each other for internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for the Impact of Crime (IOC) curriculum with all
current lessons. The youth receive separate workbooks, which are copied from the back of the
IOC manual. The curriculum also includes implementation guidelines, identifies the order the
lessons should be delivered, and includes detailed instructions for the delivery of every lesson.
IOC groups are scheduled for Mondays and Wednesdays from 7:30am to 8:30am and Tuesdays
and Thursdays from 5:00pm to 6:00pm. A review of sign-in sheets for the groups held during the
twelve-month annual review period indicated the groups were held sporadically during the month
and not on a consistent schedule. Each group was held for one hour per a review of the sign-in
sheets.
Documentation for IOC groups is maintained in a binder in the clinical director's office. The binder
includes training documentation, sign-in sheets, and fidelity monitoring forms. The binder is
divided by the cohorts for IOC. All youth are eligible to attend the IOC groups. Impact of Crime
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
groups are closed groups, meaning once the first session has been completed no youth may be
added to the cohort.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on July 29, 2021 from 12:00pm to 1:00pm. There was
one staff, one facilitator, and seven youth present. The youth are on chapter three, Victim Impact
Statement. The documentation reviewed showed two facilitators assigned to this group; however,
the primary facilitator was conducting the group alone until she left on medical leave for thirty (30)
days. The second facilitator conducted the group in her absence and when she returned, he was
no longer in the group as a facilitator.
The facilitator was comfortable teaching the lesson. She was able to use her own words and
relate the lesson to the youth in a way they would understand. One youth did not participate for
the first part of the lesson. During the second half of the lesson the youth was disruptive and was
provided a warning about being able to stay in group. The facilitator and staff both addressed the
youth during the session. Although he was disruptive, the group was able to continue and the
other youth did not allow the disruptive youth to stop them from engaging in the lesson. The
facilitator continually used reflections and feedback to the youth throughout group. The youth
volunteered to read and all, except one, were fully engaged in the group and the discussion.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol is for the Impact of Crime (IOC) groups to be co-facilitated when possible.
Should the program experience turnover, the co-facilitator would continue with the group. If the
group was not co-facilitated, another trained staff member would continue with group facilitation.
Best practice for the program is for the employee to leave employment after the cohort has
finished, if possible.
The program did experience turnover during the annual review period which affected the IOC
groups. There was a gap in service from April 4, 2021 to November 25, 2021. This gap was due
to the facilitator falsifying the sign-in sheets for the cohort from March 22, 2021 to June 14, 2021.
This incident has been reported and rectified as of the annual review. There was an additional
cohort that began on January 14, 2021 and ended on January 24, 2021. There were two youth in
the cohort. Per the program director, the group ended due to the youths' behaviors in the group.
These two youth did not successfully complete the IOC curriculum.
The program completed six cohorts during the annual review period.
The first cohort started December 13, 2020 and ended on April 4, 2021. Six youth started and
successfully completed the group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The second cohort started November 25, 2020 and ended March 18, 2021. Six youth started and
successfully completed the group.
A third cohort started June 22, 2021 and ended on October 14, 2021. The group started with six
youth and ended with eight youth per a review of the sign-in sheets. This reviewer spoke with a
facilitator about the addition of two youth. Two youth were moved to this group due to security
concerns and one youth changed dorms due to security concerns. One facilitator was conducting
the groups until October 7, 2021. The co-facilitator became the primary facilitator on October 12,
2021. The case manager conducted the final group of this cohort.
A fourth cohort started June 22, 2021 and ended on September 22, 2021. This group started with
five youth; four youth successfully completed the group. One youth was added to
this group from another group due to safety and security concerns. One youth left the group on
September 18, 2021 and was unsuccessful in completion. There was no explanation as to why
this youth left the group.
A fifth cohort was started August 4, 2021 and ended on January 19, 2022. The group started and
ended with five youth. A new facilitator took over the group on December 1, 2021 due to the
previous facilitator leaving employment. This group was not co-facilitated. The group was held two
days per week until November 10, 2021 and then was held one day per week through the end of
the review.
A sixth cohort was started on January 27, 2022 and was continuing to be held at the time of this
review.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process is for the trained facilitators to monitor each other for internal fidelity
monitoring observations of Impact of Crime. There were nine total staff who facilitated Impact of
Crime (IOC) groups during the twelve month annual review period.
Internal fidelity monitoring forms are maintained in the SPEP binder kept in the clinical director's
office. A review of the fidelity forms showed there were four months of missing fidelity for one
facilitator and two months missing for another facilitator. Twenty-three out of twenty-nine internal
fidelity monitoring observations were completed during the annual review period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring. The regional compliance manager addresses any significant issues or
corrective actions with the facilitators for Impact of Crime (IOC). There is not a case management
supervisor at the program currently; when the case management supervisor position is filled, they
will be responsible for addressing corrective actions.
All internal fidelity monitoring forms were reviewed for the annual review period. There were no
significant issues or corrective actions noted on the fidelity forms. This reviewer did see marked
improvement with the fidelity notes during the fourth quarter of the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluation forms include a specific section for group
facilitation. All groups the staff are trained in were listed on the performance evaluation.
Three of the ten trained Impact of Crime (IOC) facilitators had received their annual performance
evaluations at the time of the twelve-month review. One staff no longer facilitates IOC. Three of
the staff are no longer employed at the program and did not receive an annual performance
evaluation. One trained staff has not been at the program long enough to have received an
annual performance evaluation. The regional compliance manager is not a regular facilitator for
IOC.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 29 youth in the
sample, 55% (16 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 29 youth in the
sample, 66% (19 of 29) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 13 youth
High = 9 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 31%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (25 of 29) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 31% (9 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 63 85 74%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
31 youth completed the program during the review period. Of those youth, 29 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 29 total youth sampled, only
21 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 6
and 35 weeks of service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 6 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, only 21 included dosage in the EBS Module. Of those youth with correct dosage, 19
received at least the recommended hours of service. Youth in the sample completed between 14
and 35 hours of service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups, there are no unnecessary gaps in
service, and fidelity monitoring is completed every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 WAGC IOC.pdf

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Period: March 15, 2022 - March 14, 2023
SPEP Review Conducted: March 22, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Aggression Replacement Training (ART) as a primary service for youth
identified with aggression issues through the Residential Assessment for Youth (RAY). The
therapeutic managers are trained to facilitate ART groups.
During the annual review period there were seven trained staff: the regional clinical director, one
clinical director, and five therapuetic managers. Five of the staff facilitated ART groups during the
annual review period. All training records were provided for review and the staff was formally
trained in the ART curriculum. The regional clinical director conducts internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for the Aggression Replacement Training (ART)
curriculum, authored by Barry Glick and John Gibbs, third edition, revised and expanded. The
manual includes all lessons. Each lesson has facilitation guidelines throughout the book. There
are no separate manuals for the youth in ART groups.
Aggression Replacement Training is held on Tuesdays, Wednesdays, and Thursdays. The times
varied on the groups in each cohort. The program maintains all documentation for ART in a binder
kept in the clinical director's office. Documentation in the binder includes sign-in sheets, internal
fidelity monitoring forms, and the schedule for each cohort. The cohorts for ART are closed,
meaning no youth should be added to the group once the first lesson has been completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Aggression Replacement Training (ART) was conducted on Wednesday,
December 7, 2022 from 2:00pm to 3:00pm. There were five youth, one facilitator, and one staff in
group. The facilitator provided a copy of the lesson to follow. The topic for the day was Anger
Control Training, week 23: Rehearsal of Full Sequence. The facilitator did not follow the script and
did not conduct the role plays for the Rehearsal of Full Sequence. The facilitator did have the
youth complete hassle logs and collected them, which has been an issue in previous groups.
However, the facilitator did not review the hassle logs with the group. The facilitator reviewed all of
the seven steps with the youth, allowed a volunteer to create a posterboard with the steps. The
group discussed all the meanings of the steps. Each youth was provided a painting canvas and
could pick one of the seven steps. The youth were to come up with examples for each of the
steps they chose. The canvases will be posted in the program. The facilitator did not facilitate the
group as designed. It is a requirement for the lesson for the youth to conduct role plays and
demonstrate the understanding of the Full Sequence.
The reviewer discussed the issues with the regional clinical director on Thursday, December 08,
2022. The facilitator did not conduct this lesson properly. The regional clinical director will be
discussing the facilitation issues with the program director.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Aggression Replacement Training
(ART), is for another trained staff to continue with the cohort until completed. The program did
experience turnover during the annual review period. Three therapuetic managers left
employment. One staff left during a cohort and it was not being co-facilitated. Two others left in-
between cohorts.
There was one gap in service of thirty-three (33) days between cohorts. There was no reason for
the gap provided by the program.
The program completed three cohorts during the annual review period. The first cohort began on
April 5, 2022 and ended on June 9, 2022. The cohort began with three youth and two youth
successfully completed the curriculum. One youth was arrested during the cohort and did not
return to the program.
The second cohort began on July 5, 2022 and ended on September 8, 2022. The group started
with six youth and six youth successfully completed the curriculum. One youth was out on
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
transport during groups held September 23rd and September 24th. Make-up groups were
completed and sign-in sheets were reviewed.
The third cohort began on October 11, 2022 and ended on January 5, 2023. Four youth began
the group and five youth successfully completed the curriculum. One youth missed three lessons
due to arriving at the facility shortly after the cohort began. The facilitator conducted three
individual sessions with the youth and added the youth to the group on October 18, 2022. One
lesson was repeated due to the facilitator believing the youth did not completely understand the
concepts in the lesson. The original facilitator was removed from the group on December 15,
2022 due to not completing the curriculum with fidelity. Another trained staff continued with the
cohort. There was a short gap of in service of nineteen (19) days during the cohort between
December 15th and January 3rd.
One cohort was being held during the time of the review. This cohort began on January 31, 2023
and was still being held at the time of the review. This cohort is being co-facilitated. The group
began with ten (10) youth and nine youth were still in the group at the time of the review. One
youth was removed from group after refusing to attend three sessions. Another youth refused
group on February 14th and 15th. Another youth refused group on March 14th. A different youth did
not attend group on March 16th and no explanation was provided.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The regional clinical director has conducted internal fidelity monitoring for Aggression
Replacement Training (ART) groups during the annual review period. There were five staff who
facilitated the groups.
The requirements for fidelity monitoring of ART is for each facilitator to be monitored two times
during each month of facilitation. Two internal fidelity monitorings were missing for two facilitators
in January 2023.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy for corrective actions of facilitators for Aggression Replacement Training
(ART) is for the regional clinical director and program director to address the issues with the
facilitator and document all issues and results.
During the third quarter of the annual review period the clinical director did not facilitate the
observed ART group with fidelity. A corrective action was taken by the regional clinical director
regarding the issues. This facilitator cannot facilitate ART groups until formally re-trained with
fidelity. According to a review of the sign-in sheets, this facilitator has not been running groups
since that time. The regional clinical director reported this facilitator is signed up for the training,
but it had not taken place at the time of the review.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluations have an addendum which addresses the primary
services the staff facilitate. Three trained staff were no longer employed at the program and did
not receive an annual performance evaluation. One of the three facilitators remaining did receive
an annual performance evaluation on January 29, 2023. The second facilitator received a ninety
(90)-day performance evaluation on March 21, 2023. The third facilitator has not been employed
long enough to have received an evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 18 youth in the
sample, 100% (18 of 18) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 18 youth in the
sample, 100% (18 of 18) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 1 youth
High = 14 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 78%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (17 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 78% (14 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 88 100 88%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
43 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Duration. Of the 18 total youth sampled, all
18 included dosage with end dates in the EBS Module. Of those youth, 18 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 16 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, all 18 included dosage in the EBS Module. Of those youth, 18 received at least the
recommended hours of service. Youth in the sample completed between 30 and 44 hours of
service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring the facilitator adheres to the manual, facilitator turnover does not impact
groups, there are no unnecessary gaps in service, and fidelity monitoring is completed every
month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 WAGC ART.pdf


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Period: March 15, 2022 - March 14, 2023
SPEP Review Conducted: March 22, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service for all youth in the program. The
case management staff and direct care staff are trained to facilitate IOC groups. There have been
six trained staff during the annual review period: the regional compliance director, the clinical
director, two transition managers, one director care staff, and one shift supervisor. All training
records were provided for review.
Of the six staff trained, four staff have facilitated groups during the annual review period. The
trained staff either observe each other or the regional compliance director conducts internal
fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for the Impact of Crime (IOC) curriculum with all
current lessons. The youth receive separate workbooks, which are copied from the back of the
IOC manual. The curriculum also includes implementation guidelines, identifies the order the
lessons should be delivered, and includes detailed instructions for the delivery of every lesson.
The schedule has changed multiple times throughout the annual review period and held at
different times of the day. The documentation for IOC groups is maintained in a binder kept in the
clinical director's office. Impact of Crime groups are closed groups meaning youth should not be
added to a cohort once the first lesson has been completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This reviewer attempted to observe Impact of Crime groups throughout the annual review period.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process in the event of facilitator turnover for Impact of Crime (IOC) groups is for
another trained staff to continue with group facilitation. The program did experience IOC facilitator
turnover during the annual review period.
There was a gap in service of forty-nine (49) days from January 15, 2023 to March 5, 2023. An
email received from the regional clinical director indicated the gap was due to the program staff
needing time to assess the youth who were transferred into the program and were still needing
IOC services. According to the regional clinical director, the youth are in an "accelerated" IOC
group to ensure they are able to complete the curriculum before graduating. There was also a
short gap of three weeks, August 14, 2022 to September 6, 2022 due to the youth and staff being
in quarantine.
The program completed six cohorts of IOC during the annual review period. The first cohort
started January 27, 2022 and ended on June 14, 2022. The cohort began with nine youth and
four youth successfully completed the curriculum. The group was co-facilitated from January until
the end of March. Two of the youth were reassigned to a different group after the first lesson.
Three youth were released from the program prior to completion. These youth did not successfully
complete the curriculum. These groups were held sporadically during this first quarter and not on
a consistent schedule.
The second cohort started January 24, 2022 and completed on April 25, 2022. This group was not
co-facilitated. The group started with seven youth and ended with four youth. Three youth were
released from the program prior to completion. These youth did not successfully complete the
curriculum. Another youth was released one week prior to the group completion, however this
youth had one-on-one sessions prior to release and was able to successfully complete the IOC
group.
The third cohort started on April 4, 2022 and ended July 29, 2022. The group began with six youth
and five youth successfully completed the group. One youth left the program on July 10, 2022.
There were five individual sessions with the youth prior to leaving the program. This youth was
able to successfully complete the curriculum. According to a review of the sign-in sheets, times
were missing on July 20, 2022 and July 25, 2022. The rest of the sign-in sheets show the groups
were held at inconsistent times.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The fourth cohort started on June 3, 2022 and ended on September 25, 2022. The group began
with seven youth and six youth successfully completed the curriculum. One youth was removed
from the group and completed in a different cohort.
The fifth cohort started on July 12, 2022 and ended on December 6, 2022. The group began with
eleven (11) youth and nine youth successfully completed the curriculum. There was a three week
gap during this group due to the youth being in quarantine during that timeframe. One youth
graduated from the program early and completed individual sessions conducted by the facilitator.
One youth was ended on September 29, 2022 due to going to jail and not returning.
The sixth cohort began on September 20, 2022 and ended on January 15, 2023. The group
began with seven youth and seven youth successfully completed the curriculum. There were two
lessons provided out of order. According to a review of the sign-in sheets the fifth objective for
chapter four was completed on January 15, 2023 from 11:00am - 12:00pm and the fourth
objected of chapter four was completed on January 15, 2023 from 12:00pm to 1:00pm.
A seventh cohort is currently being held at the program. The group began on March 5, 2023 and
was still being held at the time of the review. Ten (10) youth started the cohort and ten (10) youth
were on the sign-in sheet as of March 14, 2023.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the trained staff to monitor each other or the regional compliance
director conducts internal fidelity monitoring for Impact of Crime (IOC) facilitators. There were four
IOC facilitators during the annual review period. One observation was missing from July 2022. All
other observations were completed.
The internal fidelity forms included detailed feedbak for each of the facilitators. The regional
compliance director conducted all observations.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The regional compliance director and program director address any signficant issues or corrective
actions with the facilitators of Impact of Crime. There were no documented corrective actions for
the facilitators during the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations include an addendum which describes the primary services
the staff are trained in and facilitate. Three of the four staff had received annual performance
evaluations and each had documentation of Impact of Crime facilitation. The fourth facilitator is no
longer employed and did not receive an annual performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 35 youth in the
sample, 71% (25 of 35) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 35 youth in the
sample, 94% (33 of 35) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 5 youth
High = 21 youth
Total Youth in Sample = 35 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 60%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 21 youth
Total Youth in Sample = 35 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (30 of 35) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 60% (21 of 35) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
43 youth completed the program during the review period. Of those youth, 35 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 35 total youth sampled, only
33 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 25
received at least the recommended weeks of service. Youth in the sample completed between 8
and 25 weeks of service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 35 total youth
sampled, all 35 included dosage in the EBS Module. Of those youth, 33 received at least the
recommended hours of service. Youth in the sample completed between 15 and 44 hours of
service, with an average of 32 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring there are no unnecessary gaps in service and fidelity monitoring is completed
every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 WAGC IOC.pdf