summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Seeking Safety
SPEP Review Period: March 15, 2022 - March 14, 2023
SPEP Review Conducted: March 22, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seeking Safety as a primary service for youth in the substance abuse
program. The therapy staff are trained to facilitate Seeking Safety groups. During the annual
review period six staff were trained in the Seeking Safety curriculum: four therapuetic managers,
one clinical director, and the regional clinical director. All training records have been provided for
review.
Five therapists have facilitated Seeking Safety groups during the annual review period. The
regional clinical director conducts internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for Seeking Safety. Chapter two of the manual
provides all instruction information for facilitating the curriculum. All lessons and worksheets are
located in the manual. The manual is kept in the clinical director's office. There are not separate
student manuals.
Seeking Safety groups are held on Sundays and Mondays. This schedule has changed during the
annual review period from Fridays and Saturdays to Wednesdays and Thursdays to Mondays and
Fridays and now on Sundays and Mondays.
Documentation for Seeking Safety groups is maintained in a binder kept in the clinical director's
office. The binder includes sign-in sheets and internal fidelity monitoring. Seeking Safety groups
are held as open groups, meaning youth can join upon admission to the program and remain in
the group until successful completion of the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation was scheduled for Friday, December 9, 2023. However, the group was cancelled
on that date and an observation was not conducted.
No observation of Seeking Safety was conducted during the annual review period; therefore, this
indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of facilitator turnover for Seeking Safety is for other trained
staff to facilitate the group. The program rotates facilitators so the youth are familiar with all of
them. The program did experience facilitator turnover during the annual review period. Three staff
left employment during the first and third quarters.
All sign-in sheets were provided for review. There were no gaps in service during the annual
review period. All groups had no more than fifteen (15) youth in attendance at a time.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The regional clinical director has conducted internal fidelity monitoring for Seeking Safety during
the annual review period. Five therapuetic managers have facilitated these groups.
All internal fidelity monitoring forms were provided for review. There were no missing observations
during the annual review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The regional clinical director and program director address any significant issues or corrective
actions for the facilitators of Seeking Safety. All internal fidelity forms include a section to
document issues and corrective actions. According to a review of the forms, there were no
corrective actions during the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluations have an addendum which addresses the primary
services the staff facilitate. Of the five therapeutic managers who facilitated Seeking Safety
groups during the annual review period, two of them had performance evaluations. Three of the
staff left employment before performance evaluations were completed. One evaluation was
completed on January 29, 2023 and a ninety (90)-day evaluation was completed on March 21,
2023.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 100% (19 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 100% (19 of 19) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 12 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 63%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (19 of 19) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 63% (12 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 95 95 100%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
43 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 19 total youth sampled, all
19 included dosage with end dates in the EBS Module. Of those youth, 19 received at least the
recommended weeks of service. Youth in the sample completed between 29 and 85 weeks of
service, with an average of 49 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, all 19 included dosage in the EBS Module. Of those youth, 19 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 44 and 98 hours of
service, with an average of 65 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can maintain their SPEP Quality of Service Delivery
score by continuing the practices in place at the time of this review.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 WAGC SS.pdf

service_type: Seeking Safety