summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
Defuniak Springs, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Period: April 1, 2023 - March 31, 2024
SPEP Review Conducted: April 9-12, 2024
Florida Department of Juvenile Justice
Report Date: 9/30/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggrssion Replacement Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program uses Aggression Replacement Training (ART) for their primary service for youth
identified wth aggression risk factors through the Residential Assessment for Youth (RAY).
Aggression Replacement Training (ART) facilitator training is provided by a qualified trainer over a
three-day period. There were eight total staff trained in ART. The program previously had two
therapeutic managers and a clinical director with ART certifications. The program currently has
two therapeutic managers, one clinical director, and two regional clinical compliance specialists.
The two therapeutic managers are the primary facilitators for the program.
Fidelity monitoring is completed mainly by the clinical director and the regional clinical compliance
specialists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Aggression Replacement Training (ART) curriculum, authored by Barry
Glick and John Gibbs, third edition, revised and expanded. The manual includes all lessons. Each
lesson has facilitation guidelines throughout the book. There are no separate manuals for the
youth in ART groups.
Aggression Replacement Training is held on Tuesdays, Wednesdays, and Thursdays. The
program maintains all documentation for ART in a binder kept in the clinical director's office.
Documentation in the binder includes sign-in sheets, internal fidelity monitoring forms, and the
schedule for each cohort. The cohorts for ART are closed, meaning no youth should be added to
the group once the first lesson has been completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Specialist observed a session of Aggression Replacement Training (ART)
on April 9, 2024 at 2:02pm. There were eight youth, two staff, and two facilitators present. The
facilitators began group with the review of group rules and had a youth do group check-in. The
facilitators reviewed the previous lession, reviewed and discussed steps to lesson, several
examples of role-play were completed by the youth and steps were reviewed with each youth for
feedback. The group closed with the facilitator confirming the youth did a excellent job. She asked
the youth to respond with what lession will be reviewed next, and assigned homework. The group
ended at 2:50pm. The program uses handwritten documentsation that is filed in a curriculum
specific binder.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Aggression Replacement Training
(ART), is for another trained staff to continue with the cohort until completed. The service is co-
facilitated when possible, to avoid interruption in service delivery. The program had two facilitators
leave in the middle of a cohort during the annual review period. A certified facilitator took over in
both circumstances. There were no gaps in service during this quarterly review period. Three
cohort were completed during the review period. Five cohorts of ART were completed during the
annual review period.
A cohort began on January 31, 2023 and ended on April 6, 2023. The cohort started with ten (10)
youth and ended with nine youth.
A cohort began on May 16, 2023 and ended on July 20, 2023. The cohort started with ten (10)
and ended with ten (10) youth.
A cohort began on August 15, 2023 and ended on October 26, 2023. The cohort started with
seven youth and ended with five completing the group. Two youth refused and were dropped from
the group on August 29, 2023 and October 4, 2023. There was a gap in service within the cohort
from August 31 to September 12, 2023 which the program could not explain the reason. One
youth refused a session and received a timely make-up session.
A cohort began on November 28, 2023 and ended on February 1, 2024. The cohort started with
ten (10) youth and ended with nine completing the group. On December 8, 2023, one youth went
into detention and was dropped from the group. Three incidents were observed where a youth
missed a session and received a timely make-up session.
A cohort began on February 13, 2024 and went to the end of the review period of March 31, 2023.
The cohort started with ten youth and ended with ten (10) youth currently in group. The cohort
completed lesson four twice on February 20 and 27, 2024. Lesson seven was skipped. There
were two occassions when two youth missed a session with no make-up session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for fidelity monitoring to be completed by a trained Aggression
Replacement Training (ART) facilitator. Review of the fidelity monitoring completed during the
review period was done by either the two regional clinical compliance specialists or the clinical
director.
The cohort from August 15, 2023 to October 26, 2023, revealed one facilitator received the
necessary fidelity monitoring for the month of August and September 2023. The other facilitator
was reported to be on leave during those months and did not receive fidelity monitoring. Both
facilitators received the necessary fidelity monitoring for October 2023.
The cohort from November 28, 2023 to February 1, 2024 did not have a Skillstreaming fidelity
monitoring for the month of January 2024.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring of Aggression Replacement Training (ART) facilitators. If a corrective
action is identified, the regional clinical compliance specialist and clinical director is to address the
issues with the facilitator and document all issues and results. Interview with the clinical director
states all signficant issues or corrective actions are addressed with the facilitators of Aggression
Replacement Training (ART). Her process of addressing any corrective actions is to approach the
facilitator directly after the group when the problem is observed and readdress it again in the next
observation.
The clinical director reports no corrective actions were taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program conducts annual evaluations at the annual date of hire. Both therapuetic managers
and the clinical director provided their annual evaluations for review, which included the facilitator
skill delivering the intervention.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 24 youth in the
sample, 100% (24 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 24 youth in the
sample, 100% (24 of 24) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 71%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 3 youth
Moderate-High = 6 youth
High = 8 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 8 youth
Total Youth in Sample = 24 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 71% (17 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 33% (8 of 24) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 15 25 60%
Totals 90 100 90%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
40 youth completed the program during the review period. Of those youth, 24 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 24 total youth sampled,
only 24 included dosage with end dates in the EBS Module. Of those youth with correct dosage,
24 received at least the recommended weeks of service. Youth in the sample completed between
10 and 13 weeks of service, with an average of 10.54 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, only 24 included dosage in the EBS Module. Of those youth with correct dosage, 24
received at least the recommended hours of service. Youth in the sample completed between 30
and 30 hours of service, with an average of 30 hours.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring fidelity monitoring is completed every month for each facilitator.
Walton Academy for Growth and Change can maintain their SPEP Amount of Service score by
continuing to ensure dosage for all youth is recorded accurately in EBS and by continuing to
ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

report_type: SPEP

file_name: FY2023-24 Walton Academy for Growth and Change ART.pdf

program_status_note: Open

service_type: Aggression Replacement Training


state: FL

facility_name: Walton Academy for Growth and Change

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
Defuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Period: April 1, 2023 - March 31, 2024
SPEP Review Conducted: April 9-12, 2024
Florida Department of Juvenile Justice
Report Date: May 16, 2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program uses Impact of Crime (IOC) as a primary service. The case management and direct
care staff are trained to facilitate IOC groups. There are fifteen (15) total staff trained to facilitated
IOC during the review period. The trained staff are two coach counselors, one supervisor, four
transitional service coordinators, one cook, two regional compliance specialists, and the director
of student services. The trained staff who have left the program are one clinical director, two
transitional service coordinators, and one coach counselor. Case Management and direct care
staff are the primary facilitators for the program. All training records were provided for review.
Any trained IOC facilitator can conduct internal fidelity monitoring for IOC groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual for the Impact of Crime (IOC) curriculum, with all current lessons.
The youth receive separate workbooks, which are copied from the back of the IOC manual. The
curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered, and includes detailed instructions for the delivery of every lesson.
The schedule changes throughout the review period and held at different times of the day,
depending on the facilitator’s schedule. Groups are held twice a week for one hour a session, with
at least one day between. The documentation for IOC groups is maintained in a binder kept in the
clinical director's office. Impact of Crime groups are closed groups meaning youth should not be
added to a cohort once the first lesson has been completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Specialist observed a session of Impact of Crime (IOC) on April 10, 2024 at
2:02pm. There were seven youth, two staff, and one facilitator present. The facilitator began
group with the review of chapter four objective four and had the youth check-in. The group was
conducted with two exercises, a video, group discussion, and the facilitator charted the action
steps while eliciting youth participation. The facilitator was knowledgeable of the material and
provided it in a way the youth understood. Rules of the group were reviewed at the end of the
group which was 2:59pm.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of facilitator turnover for Impact of Crime (IOC) is for another
trained facilitator to continue with group facilitation. The program had four facilitators leave with
two facilitators leaving in the middle of a cohort. A certified facilitator took over in both
circumstances. There were no gaps in service. There were a total of eight cohorts completed for
annual review period.
The cohort started on March 5 and ended on April 20, 2023. The cohort started with ten (10)
youth and ended with eight youth. Two youth were removed at the start of chapter two for refusal
to attend.
The cohort started on April 11 and ended on August 28, 2023. The cohort started with eight youth
and ended with six youth. Two youth were removed for behavioral concerns.
A cohort started on April 12, 2023 and ended on August 13, 2023. The cohort started with eight
youth and ended with four youth completing the group. On April 16, 2023, one youth was removed
for refusing treatment. On June 2, 2023, two youth were arrested and one youth was released
from the program on August 7, 2023. The facilitator left on June 28, 2023 and another facilitator
resummed the group July 12, 2023. Under the new facilitator, groups were held in the evening for
the month of August 2023.
A cohort started on July 11, 2023 and ended on November 14, 2023. The cohort started with
twelve (12) youth and ended with twelve (12) youth completing the group. On September 5, 2023,
the group sign-in sheet did not have a start date and the lesson was marked twice. There were
two groups held on two consecutive days October 19 and 20, 2023. In November, two sessions
were completed out of order with chapter 7 objective 4 done before objective 3.
A cohort started on August 22, 2023 and ended on January 5, 2024. The cohort started with eight
youth and ended with five youth completing the group. On September 7, 2023, one youth was
added to the group. On October 5, 2023, one youth was removed for refusing treatment and one
youth was arrested on October 10, 2023. Two youth were later removed from group for several
refusals on November 14, 2023.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A cohort started August 13, 2023 and ended on December 23, 2023. The cohort started with
seven youth and ended with four youth completing the group. Two youth were arrested on
October 9, 2023 and one youth was released from the program on Octover 21, 2023. For the
month of August the facilitator continued to put the date of the session instead of the cohort's start
date on the sign-in sheets.
A cohort started November 29, 2023 and went to the end of the review period of March 31, 2024.
The cohort started with nine youth and ended with nine youth currently in the group. One youth
was arrested on December 6, 2023 and another youth was added on December 9, 2023. One
youth started late and receive three make-up sessions and one youth was received early and
received two make-up sessions.
A cohort started February 11, 2024 and went to the end of the review period of March 31, 3024.
The cohort started eight youth and ended with seven youth currently in the group. One youth was
arrested on February 11, 2024. One youth was arrested and returned with a make-up session.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for the trained staff to monitor each other and conduct internal fidelity
monitoring for Impact of Crime (IOC) facilitators. There were eleven (11) IOC facilitators during
the annual review period. The internal fidelity forms included detailed feedback for each of the
facilitators.
There was one missing internal fidelity monitoring for one facilitator in June 2023, and one
missing in July 2023 for a second facilitator. The cohort from November 29, 2023 to the end of the
review period of March 31, 2024. One November fidelity monitoring was missing for both
facilitators on the November 29, 2023 session.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The director of student services addresses any signficant issues or corrective actions with the
facilitators of Impact of Crime (IOC). Her process of addressing any corrective actions is to
approach the facilitator directly after groups when the problem is observed and readdress it again
in the next observation.
The director of student services reported no corrective actions for the facilitators during the annual
review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The programs protocol for evaluating staff is that the staff get annual evaluations around their
date of hire. Of the five applicable staff providing Impact of Crime cohorts, two staff were
applicable to have received an evaluation for the annual review period. Neither staff had an
annual evaluation completed for the annual review period.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 40 youth in the
sample, 43% (17 of 40) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 40 youth in the
sample, 73% (29 of 40) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 65%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 7 youth
Moderate-High = 8 youth
High = 11 youth
Total Youth in Sample = 40 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 28%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 11 youth
Total Youth in Sample = 40 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 65% (26 of 40) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 28% (11 of 40) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 13 25 52%
Totals 63 85 74%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
40 youth completed the program during the review period. Of those youth, 40 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 40 total youth sampled, only
27 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended weeks of service. Youth in the sample completed between 7
and 23 weeks of service, with an average of 15 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 6 points for Amount of Service: Contact Hours. Of the 40 total youth
sampled, only 40 included dosage in the EBS Module. Of those youth with correct dosage, 29
received at least the recommended hours of service. Youth in the sample completed between 12
and 33 hours of service, with an average of 27 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring fidelity monitoring is completed every month for each facilitator and each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

file_name: FY2023-24 Walton Academy for Growth and Change IOC.pdf

program_name_raw: Walton Academy for Growth and Change

service_type: Impact of Crime


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
Defuniak Springs, Florida 32435
Primary Service: Seeking Safety
SPEP Review Period: April 1, 2023 - March 31, 2024
SPEP Review Conducted: April 9-12, 2024
Florida Department of Juvenile Justice
Report Date: 9/18/2025
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program uses Seeking Safety for their primary service for youth in the substance abuse
program. There is a training protocol which therapeutic managers are provided training by the
clinical director and/or regional clinical team which can be completed in person or over Teams.
Therapeutic managers are trained to facilitate Seeking Safety groups. During the review period,
four staff were trained to provide the service. Two therapeutic managers, the clincial director, and
the previous clinical director facilitated Seeking Safety groups. The program clinical director and
the two regional clinical compliance specialist provided monthly internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual for Seeking Safety. Chapter two of the manual
provides all instruction information for facilitating the curriculum. All lessons and worksheets are
located in the manual. The manual is kept in the clinical director's office. There are not separate
student manuals.
Seeking Safety groups are held twice a week for one hour a session. All youth in the Substance
Abuse Overlay component of the program are placed in Seeking Safety groups. Documentation is
maintained in a binder in the clinical director's office. The binder includes sign-in sheets and
internal fidelity monitoring. Seeking Safety groups are held as open groups, meaning youth can
join upon admission to the program and remain in the group until successful completion of the
program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Specialist observed a session of Seeking Safety on April 58, 2024 at
2:02pm. There were twelve (12) youth, two staff, and one facilitator present. The facilitator began
with announcing the group topic, had the group review rules, and did a group check. A handout
was distributed and discussed with youth feedback. Youth participated in role play and the group
concluded at 3:47pm with a homework assignment.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Seeking Safety, is for another trained
staff to continue with the cohort until completed. The service is co-facilitated when possible, to
avoid interruption in service delivery. The program did not have any facilitator turnover or gaps in
service for the annual review period.
On September 11, 2023, fifteen (15) youth signed in with an additional three youth signed in for
supportive sevices. On September 17 & 18, 2023, fifteen (15) youth signed in with an additional
four youth signed in for supportive sevices. On September 24, 25, & 28, 2023, fifteen (15) youth
signed in with an additional five youth signed in for supportive sevices. October 2, 2023, one
youth received supportive services with no make-up session. On October 9, 2023, two youth were
off campus with no make-up session. On November 16, 2023, one youth received supportive
services with no make-up. On December 6, 2023, one youth is marked as refused with no make-
up session. On March 5, 2024, one youth refused with no make-up session.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's protocol is for fidelity monitoring to be completed by a the clinical director or either
regional clinical compliance specialists. There were three Seeking Safety facilitators for this
review period and no missing fidelity monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring of Seeking Safety facilitators. If a corrective action is identified, the
regional clinical compliance specialist and clinical director is to address the issues with the
facilitator and document all issues and results. Interview with the clinical director states all
signficant issues or corrective actions are addressed with the facilitators. Her process of
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
addressing any corrective action is to approach the facilitator directly after the group when the
problem is observed and readdress it again in the next observation.
The clinical director reports no corrective actions were taken during the review period. However,
on November 2, 2023, the facilitator did have some noted coaching on the internal fidelity
monitoring sheet review.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program conducts annual evaluations at the annual date of hire. Both therapuetic managers
and the clinical director provided their annual evaluations for review, which included the facilitator
skill delivering the intervention.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 75% (15 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 85% (17 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 70%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 3 youth
Moderate-High = 4 youth
High = 7 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 35%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 70% (14 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 35% (7 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 18 25 72%
Totals 82 95 86%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
40 youth completed the program during the review period. Of those youth, 20 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 20 total youth sampled, only
18 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended weeks of service. Youth in the sample completed between 15
and 59 weeks of service, with an average of 34 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, only 20 included dosage in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended hours of service. Youth in the sample completed between 31
and 93 hours of service, with an average of 50 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 94%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can maintain their SPEP Quality of Service Delivery
score by continuing the practices in place at the time of this review.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 Walton Academy for Growth and Change Seeking Safety.pdf

service_type: Seeking Safety