summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, Inc
(Contract Provider)
285 Gene Hurley Road
DeFuniak, Florida 32435
Primary Service: Impact of Crime
SPEP Review Date(s): 3/10/20
Florida Department of Juvenile Justice
Report Date(s): 3/10/20
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has trained therapeutic managers to conduct Impact of Crime (IOC) groups. At this
time, there are two trained therapeutic managers, a coach counselor, and the recreational
therapist who are formally trained in IOC.
There have been seven formally trained staff during the review period. All training documentation
was located in the Department's learning management system, SkillPro. This reviewer suggested
SkillPro records be put in the Standardized Program Evaluation Protocol (SPEP) binder.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program conducted Impact of Crime (IOC) groups on Mondays and Wednesdays from
3:00pm to 4:00pm. The schedule recently changed and the groups are now being held from
6:00pm to 7:00pm on Wednesdays and Fridays. The group schedule changed due to the current
facilitators' schedule. All youth in the program are eligible to attend IOC groups pursuant to the
program's contract.
Documentation for IOC groups are maintained in the Standardized Program Evaluation Protocol
(SPEP) binder maintained in the director of student services office. The program maintains the
correct manual for IOC. The youth have their own worksheets provided during group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime group was observed on Monday, December 16, 2019 from 3:00pm to 4:00pm.
Six youth were in attendance for the group. The group did a full review of the last three chapters
completed. The facilitator did an excellent job of having information ready to review. The facilitator
had a flip chart ready with all review information available. The youth participated and were
engaged throughout the group. The facilitator provided appropriate feedback and interaction with
the youth. All were provided the opportunity to participate throughout the group.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy is to have multiple staff trained to conduct Impact of Crime (IOC) groups.
During the review period, two trained staff left employment. There was a gap in service of two
months during one cohort. The reason provided for the gap was due to shortage of staff. This gap
occurred from May 21, 2019 through July 23, 2019. The previous facilitator left employment in the
end of July 2019. There were other trained staff during this gap in service.
There have been two cohorts completed during the review period. The first cohort was held from
February 26, 2019 to September 17, 2019. Upon review of the attendance logs, the facilitator was
running the groups for thirty to forty minutes only, which does not meet the dosage for IOC. When
the previous faciliator left employment, the director of student services completed the group. The
director of student services did an ongoing review of the previous chapters with the youth in the
group to ensure the concepts were taught. This reviewer spoke with the director of student
services and the facilitator feels the youth did learn the concepts and is ready to close out the
group. The previous designated mental health clinical authority (DMHCA) and director of student
services made the decision to end the class on the next group date and provide a graduation from
the group for the three youth left in the group.
The second cohort was held from October 14, 2019 to February 17, 2020. A review of the
attendance logs showed the groups were held for one hour two times a week and met the
required dosage. The group began with seven youth and ended with five youth. Upon further
review, two youth were successfully discharged from the program during this group; one in
December 2019 and one in February 2020. This issue was discussed with the director of student
services, acting DMHCA, and program director. Youth are to be able to complete IOC groups
prior to successful completion of the program. It was also discussed a system being developed to
track youth based upon entrance dates and potential release dates so all youth are able to attend
IOC.
A third cohort was started on February 26, 2020. The group started with eight youth; however, two
youth absconded from the program and were subsequently discharged.
There were no secondary groups started during the gap between February 2019 and September
2019. No other youth completed IOC during this timeframe. There are currently four trained staff
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
still employed in the program. Two are co-facilitating the current cohort. Once this cohort is
completed, each facilitator will begin a group. It was suggested there be enough groups created to
ensure all youth are able to attend an IOC group.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for the director of student services to conduct internal fidelity monitoring.
Five of twelve internal fidelity monitorings were completed during the review period. This issue
was discussed with the director of student services and the program director. It was suggested
other trained facilitators assist to conduct internal fidelity monitoring on a regular basis.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process is for the director of clinical services to address any issues or corrective
actions with the group facilitators. If the facilitator is a therapeutic manager any issues will be
addressed by the acting designated mental health clinical authority (DMHCA), who is formally
trained in the Impact of Crime curriculum.
Of the five internal fidelity forms, there were no issues noted or corrective actions noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The three facilitators who consistently facilitated Impact of Crime during the review period have
not been employed long enough to have received an annual evaluation.
The director of student services had an evaluation from 2019 while in a previous position. The
director of student services has not been in the position long enough to have received a new
evaluation. The annual evaluation reviewed does not have a specific section to address group
facilitation. The director of student services will not be consistently facilitating groups, but filling in
as needed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 24 youth in the
sample, 63% (15 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 24 youth in the
sample, 58% (14 of 24) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 96%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 14 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 58%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 24 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 96% (23 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 58% (14 of 24) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 25 25 100%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
31 youth completed the program during the review period. Of those youth, 24 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 24 total youth sampled, only
17 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended weeks of service. Youth in the sample completed between 8
and 39 weeks of service, with an average of 25 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, only 19 included dosage in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended hours of service. Youth in the sample completed between 8
and 32 hours of service, with an average of 22 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups and by ensuring fidelity monitoring is
completed every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Walton Academy for Growth and Change

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, Inc
(Contract Provider)
285 Gene Hurley Road
DeFuniak, Florida 32435
Primary Service: Seeking Safety
SPEP Review Date(s): 3/10/200
Florida Department of Juvenile Justice
Report Date(s): 3/10/20
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 5 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 25%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 3 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's goal is to have all therapy staff trained in the Seeking Safety curriculum. The
therapeutic managers are trained informally by the designated mental health clinician authority
(DMHCA) during group clinical supervisions.
Three of six therapeutic managers were trained by the regional clinician for Rite of Passage in
May 2019. Two new therapeutic managers were to be trained by the previous DMHCA. Upon
review of the clinical supervision notes, there was no documentation of this training. The program
does not have written training protocol for Seeking Safety at this time. This has been discussed
previously with both previous DMHCAs, the director of student services, and the program director.
It was clarified and recommended when new therapeutic managers are trained during weekly
supervisions, it should be documented on those attendance sheets. During this review there was
documentation of ongoing training on the weekly clinical supervision notes signed by all therapy
staff and the acting DMHCA. There is still one therapeutic manager who does not have
documentation of being trained by clinical staff. When interviewed, the therapeutic manager
stated the previous DMHCA provided videos and the manual to review. There was no
documentation of one-on-one training. However, this therapeutic manager has been attending the
ongoing trainings during weekly clinical supervisions.
This reviewer reiterated to the acting DMHCA, director of student services, and program director
all training must be documented appropriately for staff to be considered trained. The program
does have a training form which was presented for review. This form will properly document
training; however, the program also needs to create a training plan for new therapeutic managers
when hired and provide the documentation for review.
The program can earn 1 point if there is a written training protocol for the primary service all
facilitators receive formal training prior to delivering services to youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Seeking Safety is conducted for youth in the program with diagnosed substance use and/or abuse
issues. The groups are held as open groups, meaning youth begin the group when they enter the
program and work through the curriculum at their own pace. Pursuant to the contract, the group is
being held three days a week for one hour. All youth housed in the substance abuse dorm are
currently attending a Seeking Safety group. The program has recently changed the attendance
forms for the youth attending the group.
All documentation for the Seeking Safety groups is maintained in a binder kept in the director of
clinical services office, at this time. Once the program has hired a designated mental health
clinician authority (DMHCA), the documents will be maintained by this person. There is an acting
DMHCA at this time, who is a current therapeutic manager as well and is being trained in some of
the duties of a DMHCA.
The program maintains the current manual for Seeking Safety. Chapter two of the manual
provides all instruction information for facilitating the Seeking Safety curriculum.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This group was not observed during this review. This reviewer will observe a Seeking Safety
group during a future quarterly review.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Four therapeutic managers and two designated mental health clinician authorities (DMHCA) left
during this review period. When a facilitator would leave, another trained facilitator would conduct
the group. A fifth therapuetic manager is leaving employment shortly. There is currently one
trained therapeutic manager with documentation to facilitate these groups.
Upon review of the attendance forms for this review period, there were no gaps in service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for the designated mental health clinician authority (DMHCA) to conduct
internal fidelity monitoring for the Seeking Safety groups.
One out of twelve internal fidelity monitorings were completed for this review period. This was
completed in December 2019 for one of the two therapeutic managers conducting the groups.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the designated mental health clinician authority (DMHCA) to discuss
any issues or corrective actions during individual clinical supervisions and re-observe the
therapeutic manager again. If the issue continues further action will be taken. The program also
updated the internal fidelity form to include more detailed information.
Fidelity monitoring was conducted only once during the review period. Internal fidelity monitoring
must be conducted to award points in this section.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual evaluation form for therapeutic managers include a section to evaluate for group
facilitation. The current group facilitators have not yet been employed long enough to have
received an annual performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 26 youth in the
sample, 8% (2 of 26) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 26 youth in the
sample, 0% (0 of 26) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 97%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 8 youth
High = 14 youth
Total Youth in Sample = 26 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 54%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 26 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 96% (25 of 26) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 54% (14 of 26) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 5 20 25%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 60 95 63%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
31 youth completed the program during the review period.
The program scored Low for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 26 total youth sampled, all
26 included dosage with end dates in the EBS Module. Of those youth, 2 received at least the
recommended weeks of service. Youth in the sample completed between 4 and 34 weeks of
service, with an average of 8 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 26 total youth
sampled, all 26 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed 25 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring there is a written training protocol for the primary service and all facilitators
receive formal training prior to delivering services to youth; by ensuring facilitator turnover does
not impact groups; by ensuring fidelity monitoring is completed every month for each facilitator;
and by ensuring any corrective action needed is identified, applied, and demonstrated.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Walton Academy for Growth and Change

program_status_note: Open

service_type: Seeking Safety


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, Inc
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Aggression Replacement Training
SPEP Review Period: March 15, 2020 - March 14, 2021
SPEP Review Conducted: March 23, 2021
Florida Department of Juvenile Justice
Report Date: 6/18/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Aggression Replacement Training (ART) as a primary service. The training
protocol for the program is for the therapeutic staff to be trained to conduct ART. Rite of Passage
has an internal trainer for ART facilitators. There have been six staff trained during the twelve
month review period. There are currently four staff still employed at the program and trained in
ART. At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for the Aggression Replacement Training (ART)
curriculum.The program maintains the ART manual authored by Barry Glick and John Gibbs, third
edition revised and expanded. The manual includes all lessons. Each lesson has facilitation
guidelines throughout the book.
Aggression Replacement Training is held on Tuesdays, Wednesdays, and Thursdays from
3:00pm to 4:00pm per the current program schedule. Youth with aggression are able to attend
ART group per the program's contract table. This is determined by the youth's Residential
Assessment for Youth and the Youth Needs Assessment Summary. ART is held as a closed
group, meaning once the group begins, no new youth may be added. Any new admissions to the
program are placed on a waiting list and will be enrolled in the next class.
The documents for ART are maintained in a binder in the clinical director's office, which includes
sign-in sheets, training documentation, and internal fidelity monitoring forms.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training (ART) was observed on Tuesday, March 23, 2021
from 4:00pm to 5:00pm. There were three youth in this cohort. One staff member was present, as
well as the new clinical director.
The lesson for the group was Understanding the Feelings of Others. The facilitator introduced the
skill to be learned for the lesson and passed out the social skills training steps card. The facilitator
then conducted an activity that was not a part of the lesson. The activity was about making good
choices in a situation. This activity was not in the lesson plan or in the script. The objectives of the
lesson were not met. It was recommended the social skill be taught on another day to ensure the
youth are able to demonstrate the social skill.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover during the twelve-month review period. A clinical director and
one therapeutic manager left employment. The program's protocol is for another trained facilitator
to continue with the group to avoid any gaps in service. The therapeutic manager who left was not
conducting an Aggression Replacement Training (ART) group at the time employment stopped.
Therefore, it did not cause a disruption to the group.
There were two cohorts completed during the review period. The first cohort was from January 29,
2020 to June 18, 2020. All attendance sheets were provided for review. Initially there were ten
youth. As of May 12, 2020 there were five youth in group. Notes indicated two youth completed
the program, though they had not yet completed ART. A third youth was removed due to escaping
from the program and not returning. A fourth youth missed a month due to court and did not
return. And the fifth youth was dropped noting, "not ready for ART".
The second cohort completed was from October 6, 2020 to December 10, 2020. This group
started with eight youth and ended with six youth. One youth left the program prior to completing
ART group and the second youth was dropped from the group due to behaviors.
There were two significant gaps in service during the twelve month review period. The first gap
was between June 18, 2020 and October 6, 2020 and the second gap was between December
10, 2020 and March 16, 2021. Per the director of student services the gaps were due to being low
in trained staffed for ART. The youth in the program, along with the main facilitator were all in
quarantine the month of September 2020.
The program can earn 2 points if there are no unnecessary gaps in service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director or another trained facilitator conduct internal fidelity monitoring for Aggression
Replacement Training (ART) groups.
No internal fidelity monitoring was provided from March to October 2020. Per the director of
student services, this was due to therapeutic managers being out for quarantine at various times.
Due to group schedules and lack of staff on-site, internal fidelity monitoring was unable to be
completed for ART. There were times when regional Rite of Passage staff were at the program
and conducted internal fidelity monitoring for ART groups.
Out of eighteen possible fidelity monitorings for the twelve month review period, six were
completed and reviewed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
If there are issues or any corrective actions during Aggression Replacment Training (ART) groups
the clinical director addresses them during one-to-one supervisions. There were no corrective
actions noted nor taken during the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of one annual performance evaluation found a section to address facilitator performance.
This section listed all curricula the employee facilitates. The comments included any strengths
and/or weaknesses of the facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 10 weeks for this type of service. Of the 19 youth in the
sample, 16% (3 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 30 contact hours for this type of service. Of the 19 youth in the
sample, 58% (11 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 13 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 68%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 13 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 68% (13 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 25 25 100%
Totals 74 100 74%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Walton Academy for Growth and Change.
36 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Duration. Of the 19 total youth sampled, only
4 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 3
received at least the recommended weeks of service. Youth in the sample completed between 7
and 10 weeks of service, with an average of 9 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, all 19 included dosage in the EBS Module. Of those youth, 11 received at least the
recommended hours of service. Youth in the sample completed between 8 and 60 hours of
service, with an average of 26 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring the facilitator adheres to the manual, there are no unnecessary gaps in service,
and fidelity monitoring is completed every month for each facilitator
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Aggression Replacement Training


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, LLC
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Impact of Crime
SPEP Review Period: March 15, 2020 to March 15, 2021
SPEP Review Conducted: March 23, 2021
Florida Department of Juvenile Justice
Report Date: 6/18/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. Pursuant to the program's
contract, youth care workers are trained to conduct IOC. However, there are numerous staff
trained in various position at the program. There have been six trained staff during the twelve
month review period. There are currently four trained staff all four staff are actively facilitating IOC
groups. All training records were provided for review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The porgram utilizes the brand-name manual for the Impact of Crime (IOC) curriculum with all
current lessons. The youth receive separate workbooks, which are copied from the back of the
IOC manual. The curriculum also includes implementation guidelines, identifies the order the
lessons should be delivered, and includes detailed instructions for the delivery of every lesson.
Impact of Crime groups are currently held on Mondays, Wednesdays, and Fridays from 11:45am
to 12:45pm; Tuesdays and Thursdays from 5:00pm to 6:00pm; and Tuesdays and Fridays from
5:30pm to 6:30pm. One group is being held three days a week with six youth to ensure these
youth are able to successfully complete the curriculum prior to leaving the program.
Documentation, which includes sign-in sheets, group schedule and cohorts, training
documentation, and internal fidelity monitoring forms are all maintained in a binder in the director
of students services office. According to the program's contract table all youth are eligible to
attend IOC groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed virtually on October 23, 2020. There were seven
youth in the group. The group was held for one full hour. This was the facilitator's first cohort. The
facilitator utilized the curriculum, but was able to put the script into her own words. There were no
significant issues and all lesson objectives were met. One youth did become disruptive during
group time, but the facilitator handled the situation well.
This reviewer debriefed the session with the facilitator once group was over. The facilitator is
becoming more comfortable with teaching the curriculum. The facilitator did state feeling more
comfortable with being able to co-facilitate a group until becoming more comfortable with the
curriculum.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover during the twelve month review period. However, the staff
leaving did not affect the Impact of Crime (IOC) groups. When a facilitator leaves during a cohort
another trained facilitator continues the group to avoid any gaps in service. One facilitator is
currently out due to deaths in the family. The director of student services was able to complete the
cohort.
There were six cohorts completed during this review period and two cohorts currently being held.
Impact of Crime groups are typically not co-facilitated. All sign-in sheets were provided for the
twelve month review period.
The first cohort was held from February 26, 2020 to June 17, 2020. Six youth began the group
and five youth successfully completed the group. One youth was removed from the group due to
behaviors.
A second cohort was held from June 26, 2020 to October 16, 2020. Seven youth began this group
and all successfully completed. All youth in this group were in quarantine during the month of
September 2020. The group was put on pause during that time causing a gap from August 29,
2020 to September 24, 2020. Once the youth were medically cleared, the group resumed
meeting.
A third cohort was held from July 1, 2020 to October 21, 2020. Six youth began this cohort and
five youth successfully completed the cohort. One youth was released from the group due to
disruptive behaviors in group.
A fourth cohort was held from July 1, 2020 to October 31, 2020. The group started with seven
youth and all seven successfully completed the curriculum
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A fifth cohort was held from November 25, 2020 to March 18, 2020. Six youth began this group
and all six successfully completed this cohort.
A sixth cohort was started on December 15, 2020 and is continuing to be held at this time. the
group started with six youth and currently remains at six youth. The group is due to end on March
30, 2021. A seventh cohort was started on January 14, 2021 and is continuing to be held at this
time. This group has two youth attending the group. An eight cohort was started March 22, 2021
and is continuing to meet at this time. Six youth are currently attending this group.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The director of student services or another trained faciltiator conducts internal fidelity monitoring
for Impact of Crime groups.
All internal fidelity monitoring forms were provided for review. Four out of twenty-four possible
internal fidelity monitorings were missing for the twelve month review period. All internal fidelity
monitoring forms were filled out to include feedback for the facilitators.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's protocol is for the director of student services to address any issues or corrective
actions regarding facilitation of Impact of Crime groups. All forms were provided for this review
and no corrective actions were noted for the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Four annual performance evaluations were reviewed for this curriculum. There was an entry for
evaluation of the staff's performance in delivering Impact of Crime. These included any strengths
and weaknesses of the facilitator to be made aware. All facilitators were included in this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 36 youth in the
sample, 36% (13 of 36) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 36 youth in the
sample, 47% (17 of 36) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 7 youth
Moderate-High = 7 youth
High = 16 youth
Total Youth in Sample = 36 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 16 youth
Total Youth in Sample = 36 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (30 of 36) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 44% (16 of 36) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 92%
Totals 66 85 78%
This SPEP report evaluates Impact of Crime, an intervention delivered at Walton Academy for
Growth and Change.
36 youth completed the program during the review period. Of those youth, 36 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 36 total youth sampled, only
13 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 16
and 25 weeks of service, with an average of 19 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 36 total youth
sampled, only 18 included dosage in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended hours of service. Youth in the sample completed between 5
and 35 hours of service, with an average of 30 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 84%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups and fidelity monitoring is completed
every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Walton Academy for Growth and Change
Rite of Passage, LLC
(Contract Provider)
286 Gene Hurley Road
DeFuniak Springs, Florida 32435
Primary Service: Seeking Safety
SPEP Review Period: March 15, 2020 - March 15, 2021
SPEP Review Conducted: March 23, 2021
Florida Department of Juvenile Justice
Report Date: 6/18/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seeking Safety as a primary service. The program's documentation reflected
therapy staff are trained to conduct Seeking Safety groups. The training is informal and conducted
by a licensed therapist who is trained in the curriculum. The program provided a training protocol
for Seeking Safety. "Seeking Safety Training will be conducted for new Therapeutic Managers by
a Licensed Mental Health Professional who has been trained in the curriculum within the first thirty
days. This is covered over four days with two hours per day, for a total of eight hours. The trainee
will be observing an ongoing group by a trained facilitator in addition to the formal training during
this four-day period." The clinical director also provides ongoing training during weekly group
supervision meetings. The program provided all training documentation for Seeking Safety.
There have been five trained staff members during the review period. Four of the five were main
facilitators throughout the twelve month review period. However, during the last quarter there has
been one main facilitator.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the brand-name manual for Seeking Safety. Chapter two of the manual
provides all instruction information for facilitating the Seeking Safety curriculum. All lessons and
worksheets are located in the manual. The manual is kept in the clinical director's office.
Seeking safety is held on Tuesdays and Thursdays from 3:00pm to 4:00pm and 4:00pm to
5:00pm. The contract was changed during the twelve month review period. Seeking Safety went
from being held seven days a week to two days a week. The youth receive other mental health
and substance abuse curricula on the other days, according to the program schedule.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The sign-in sheets, training records, and internal fidelity monitoring forms are maintained in a
binder in the clinical director's office. The program's contract shows all youth with a substance
abuse diagnosis are eligible to receive the Seeking Safety curriclulm. This is determined through
substance abuse assessments, Residential Assessment for Youth, and the Youth Needs
Assessment Summary.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on Tuesday, March 23, 2021 from 3:00pm to 4:00pm.
There are eleven youth scheduled for the group, ten youth were present for the full group on this
date. The eleventh youth came in thirty minutes into the group session. Two coach counselors
were present during the group. The new clinical director also attended the session on this date.
The lesson presented was Life Choice Game Review. The facilitator explained the directions to
the youth and allowed time for any questions or clarification for the game. The facilitator was
prepared with a copy of the lesson. The facilitator had copies of "Safe Coping Skills" sheets to
pass out to the youth. The group rules were reviewed along with the quote for the day.
The faciltiator followed the curriculum without reading from the lesson verbatim. The facilitator
used their own words and examples. The faciltator did a good job including all of the youth in the
group and providing opportunities for the youth to engage in the lesson. Motivational Interviewing
skills were used throughout the session. There were no behavior disruptions during this group
session.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover during this twelve-month review period. The previous clinical
director and one therapeutic manager left during this review period. There was a disruption to the
group when the therapeutic manager left; however, another trained staff member continued
facilitating the group to avoid gaps in service. The youth were familiar with the therapeutic
manager who continued with the group to ensure as little disruption to the youth as possible.
All sign-in sheets were provided for review. The groups average between eight and ten youth per
session. Two groups are held on each day of the curriculum. When a youth completes the group
this should be documented on the sign-sheets.
There were no significant gaps in service; however, a review of group sign-in sheets revealed
there were gaps in dates.There were gaps in days between May 19, 2020 to June 8, 2020, June
9, 2020 to June 12, 2020, June 12, 2020 to June 16, 2020, and June 17, 2020 to June 23, 2020.
During this time the therapuetic managers were out sporadically during the review period due to
quarantine. Other therapeutic staff has been able to conduct groups.
The program can earn 2 points if facilitator turnover does not impact groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring forms were provided for the twelve-month review peirod. The clinical
director or another trained therapuetic manager conducts internal fidelity monitoring for Seeking
Safety groups.
Nine out of twenty-five internal fidelitly forms were missing for the twelve month review period.
There was an interim clinical director, who is a current therapeutic manager, during the periods of
time the program was without a clinical director. The interim clinical director conducted internal
fidelity monitoring during that time.
The internal fidelity monitioring forms are detailed and specific to the group faciltiation
requirements.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for corrective actions is for the clinical director to address any issues with
the facilitation of Seeking Safety one-on-one with the faciltiator.
According to a review of the internal fidelity monitoring forms, there were no issues or corrective
actions noted for the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitator's employee performance evaluations found an entry for evaluation of
group facilitation. This section listed all curricula the employee facilitates. The comments included
any strengths and/or weaknesses of the facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 23 youth in the
sample, 0% (0 of 23) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 23 youth in the
sample, 0% (0 of 23) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 78%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 6 youth
Moderate-High = 4 youth
High = 8 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 35%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 78% (18 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 35% (8 of 23) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 20 25 80%
Totals 70 95 74%
This SPEP report evaluates Seeking Safety, an intervention delivered at Walton Academy for
Growth and Change.
36 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 23 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 2
and 13 weeks of service, with an average of 9 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, only 14 included dosage in the EBS Module. Of those youth with correct dosage, 0
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended hours of service. Youth in the sample completed between 4
and 38 hours of service, with an average of 24 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 79%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Walton Academy for Growth and Change can optimize their SPEP Quality of Service Delivery
score by ensuring facilitator turnover does not impact groups and fidelity monitoring is completed
every month for each facilitator.
Walton Academy for Growth and Change can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9