summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Aggression Replacement Training
SPEP Review Period: 6/1/2020-5/31/2021
SPEP Review Conducted: 6/16/2021-6/17/2021
Florida Department of Juvenile Justice
Report Date: 8/12/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2020 through May 31, 2021 shows the
program utilizes Aggression Replacement Training (ART) as their primary service. There are five
staff who are trained to facilitate the primary service. All five staff facilitated the primary service
during the review period. All trained staff are therapeutic managers. ART training is provided by a
qualified trainer over a three-day period. A trained staff member conducts the monthly internal
fidelity checks.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program utilizes the Aggression Replacement Training (ART) manual. The manual is
broken down into six sections. The first section covers Skillstreaming. The second section deals
with anger control. The third section covers moral reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
use. It includes training step guides, fidelity forms and hassle logs, and the program materials.
The program does not have separate student workbooks to give out. The facilitators make copies
of the exercises before each session to pass out to the youth. ART is delivered as a closed group.
The curriculum is to be delivered to the youth over a minimum of ten weeks for thirty hours. The
program keeps all SPEP documents in a binder labeled Aggression Replacement Training. The
program provides the service three days per week for one hour each. All youth receive ART.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training (ART) was observed on 6/17/2021. There were
seven youth present for the group, along with two staff and the facilitator. The group began at
11:10a. The group was held in the classroom in the Administration building. The facilitator was
knowledgeable of the material and followed the lesson plan. The youth were disappointed they
had to complete another session. The youth thought their ART group had finished the prior week.
One youth kept verbally expressing his frustration about having to attend another ART group.
Another youth was taken to the side by a coach counselor to be spoken to about his frustration.
The last session put together all components of the anger control chain. Each youth were given
an opportunity to demonstrate how they would use the anger control chain. The facilitator would
stop and correct the youth if needed. The group ended at 12:03p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program had facilitator turnover; however, the turnover did not affect group facilitation
of Aggression Replacement Training. The documentation shows a therapeutic manager left the
program, but finished with the cohort prior to leaving the program. The program also gained a new
clinical director on 5/1/2021.
There were five cohorts completed during the annual review period. The dates of the cohorts are:
4/6/2021-current, 1/5/2021-3/25/2021, 9/14/2020-12/11/2020, 7/11/2020-8/15/2020, and a cohort
which began prior to the annual review period and ended on 6/8/2020.
Seven youth successfully completed the cohort ending 6/8/2020.
The 7/11/2020 cohort only had one youth. There was a note with the attendance sheets which
stated the youth needed to complete the primary service prior to being released.
The 9/14/2020 cohort began with nine youth and ended with five youth. The attendance sheets
showed the facilitators were not conducting the groups in the proper order.
The 1/5/2021 cohort began with six youth and ended with five youth. The attendance sheets show
the cohort began with one therapeutic manager facilitating sessions on 1/5/2021 and 1/14/2021.
Another facilitator began co-facilitating on 1/7/2021 and took over after 1/14/2021. That facilitator
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
continued facilitating groups until 2/23/2021 when another therapeutic manager took over and
finished the cohort.
The 4/6/2021 cohort began with eight youth and has eight youth at the time of the annual review.
The facilitators conducted the first four weeks in order. The facilitators then began bouncing
around and not covering each lesson in order.
The technical assistance specialist spoke with the clinical director and facilitators and stated the
groups needed to be run in order. The clinical director stated the issue is the primary service
might be scheduled on the main facilitator’s day off. The co-facilitator would then take the group
and might not know exactly where the main facilitator left off. The technical assistance specialist
stated the facilitator and co-facilitator should conduct each group together, if they are assigned to
the same cohort.
There were three gaps in in service during the review period, but all youth were provided the
service while at the program. There was a gap of over one month between the cohort which
ended on 6/8/2020 and the cohort which began on 7/11/2020. There was a gap of thirty days
between the cohort which ended on 8/15/2020 and the cohort which began on 9/14/2020. There
was a twenty-five-day gap between the cohort which ended on 12/11/2020 and the cohort which
began on 1/5/2021.
According to the program's Standardized Program Evaluation Protocol provedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, in an effort to reduce staff turnover. In addition, the program provides
positive incentives, which in their experience has shown to improve employee motivation and
dedication, leading to better employee retention. Through this policy, employees receive credit for
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed twenty-four internal fidelity monitorings of Aggression Replacement
Training. The program was missing internal fidelity checks in July 2020 and August 2020.
There were eleven fidelity checks which were completed by the co-facilitator assigned to the
cohort, not by another trained staff member. The dates of the eleven fidelity checks were
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3/2/2021, 2/16/2021, 1/7/2021, 1/6/2021, 1/5/2021, 10/22/2020, 10/21/2020, 10/19/2020,
9/24/2020, 9/23/2020, and 9/14/2020. The technical assistance specialist told the clinical director
all internal fidelity checks needed to be completed by another trained staff member who is not
assigned to the specific cohort being observed.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions needed for facilitators of Aggression Replacement Training.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manger and be addressed in accordance with the employee handbook.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed five staff evaluations. The evaluations do not have a section to
specifically address the staff’s performance in facilitating Aggression Replacement Training. The
technical assistance specialist spoke with the clinical director and the facility administrator and
explained each performance review should have the primary services the facilitator is trained in
listed and the evaluations should specifically address the staff’s proficiency in delivering each
primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 14 youth in the
sample, 86% (12 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 14 youth in the
sample, 100% (14 of 14) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (14 of 14) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (7 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 88 100 88%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Hope
Forest Academy.
16 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program earned 8 points for Amount of Service: Duration. Of the 14 total youth sampled, only
13 included dosage with end dates in the EBS Module. Of those youth with correct entries, 12
received at least the recommended weeks of service. Youth in the sample completed between 9
and 13 weeks of service, with an average of 12 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 14 received at least the
recommended hours of service. Youth in the sample completed between 30 and 60 hours of
service, with an average of 36 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, fidelity monitoring is completed every month for each
facilitator, and each facilitator’s performance evaluation includes discussion of their skill in
delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2020-21 Hope Forest ART.pdf


state: FL

facility_name: hope forest academy

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Seeking Safety
SPEP Review Period: 6/1/2020-5/31/2021
SPEP Review Conducted: 6/16/2021-6/17/2021
Florida Department of Juvenile Justice
Report Date: 8/31/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2020 through May 31, 2021 shows the
program utilizes Seeking Safety as their primary service. There are six staff who are trained to
facilitate the primary service. Four of the six trained staff facilitated groups during the review
period. The trained staff are five therapeutic managers, and one shift supervisor. The training is
conducted by the clinical director as an in-service training. All trained staff conduct the monthly
internal fidelity checks.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance
Abuse for their primary service. The manual is broken down into twenty-five sessions. Each
session explains the topic that the therapist will cover and provides handouts for the youth to
complete. The primary service can be utilized for individual treatment or group treatment. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
program does not have separate student workbooks to give out. The facilitators make copies of
the exercises before each session to pass out to the youth. The program holds the primary
service two days per week for one hour each session. The program keeps their documentation for
the primary service in a SPEP binder labeled Seeking Safety. The binder contains all attendance
sheets, internal fidelity monitoring checklists, and group schedules. The training certificates are
held by the training coordinator at the program. All youth who are admitted to the program receive
the primary service. The primary service is held as an open group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 6/16/2021. There were eight youth present for the
group along with three staff as well as the facilitator. The group began at 11:09am and was held in
the classroom in the Administration building. The facilitator was knowledgeable of the material
and followed the lesson plan. There was one youth who appeared to not be interested in the
group and was slouching in his chair. The youth did answer questions posed to him. The group
ended at 12:02p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program did have facilitator turnover; however, the facilitator turnover did not adversely
affect the delivery of Seeking Safety because the therapeutic manager who left was not
conducting groups at that time. The program also gained a new clinical director on 5/1/2021. The
primary service is an open group held year-round. There was a gap in service from 12/12/2020 to
1/8/2021 and a two week gap from 8/5/2020 to 8/19/2020.
According to the program's Standardized Program Evaluation Protocol procedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
incentives, which in their experience has shown to improve employee motivation and dedication
and leads to better employee retention. Through this policy, employees receive credit for years of
service, employment referrals, and for contributing to the positive culture, teamwork environment
and for going above and beyond normal job expectations. Each employee who receives credit for
doing these things is entered in a monthly drawing at each facility. In the event of facilitator
turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-facilitator is
not present, a new qualified facilitator for the primary service will take over conducting the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed sixteen (16) internal fidelity monitorings of Seeking Safety. All
facilitators had fidelity checks completed on them during the annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions needed for facilitators of Seeking Safety.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manger and be addressed in accordance with the employee handbook.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed five staff evaluations. The evaluations do not have a section to
specifically address the staff’s performance in facilitation of Seeking Safety. The technical
assistance specialist spoke with the clinical director and the facility administrator and explained
each performance review should have the primary services the facilitator is trained in listed and
the evaluations should specifically address the staff’s proficiency in delivering each primary
service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 14 youth in the
sample, 29% (4 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 40 contact hours for this type of service. Of the 14 youth in the
sample, 21% (3 of 14) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 100 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (14 of 14) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (7 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 25 25 100%
Totals 79 95 83%
This SPEP report evaluates Seeking Safety, an intervention delivered at Hope Forest Academy.
16 youth completed the program during the review period. Of those youth, 14 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 14 total youth sampled, all
14 included dosage with end dates in the EBS Module. Of those youth, 4 received at least the
recommended weeks of service. Youth in the sample completed between 14 and 36 weeks of
service, with an average of 22 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 3 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 24 and 50 hours of
service, with an average of 33 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service and each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

file_name: FY2020-21 Hope Forest SS.pdf

program_name_raw: Hope Forest Academy

program_status_note: Closed

service_type: Seeking Safety


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Seeking Safety
SPEP Review Period: June 1, 2021-May 31, 2022
SPEP Review Conducted: June 23, 2022
Florida Department of Juvenile Justice
Report Date: 9/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2021 through May 31, 2022 shows the
program utilizes Seeking Safety as their primary service. There were six staff who are trained to
facilitate the primary service. Five of the six trained staff facilitated Seeking Safety during the
annual review period: one training and compliance coordinator, the clinical director, and four
therapeutic managers. One of the therapeutic managers who was trained in the primary service is
now at another program and did not facilitate any groups this review period. The training is
conducted by the clinical director or regional clinical director as an in-service training. All trained
staff are able to conduct the monthly internal fidelity checks. Four trained staff members (clinical
director and three therapeutic managers) conducted monthly internal fidelity checks during the
annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance
Abuse for their primary service. The manual is broken down into twenty-five (25) sessions. Each
session explains the topic the therapist will cover and contains handouts for the youth to
complete. The primary service can be utilized for individual treatment or group treatment. The
program does not have separate student workbooks to give out. The facilitators make copies of
the exercises before each session to pass out to the youth.
Seeking Safety groups are held on Tuesdays and Saturdays for one hour each session. The
program keeps their documentation for the primary service in a SPEP binder labeled "Seeking
Safety". The binder contains all attendance sheets, internal fidelity monitoring checklists, and
group schedules. The training certificates are held by the training coordinator at the program. All
youth who are admitted to the program receive the primary service. Seeking Safety is held as an
open group run year-round.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 6/22/2022 via Microsoft Teams. There were fifteen
(15) youth, three staff, and the facilitator present. The group was held in the education room in the
Administration building and began at 11:00a. The facilitator was knowledgeable of the material
and followed the lesson plan. The facilitator was able to redirect the youth when they were getting
off topic. The facilitator stated there was one youth who the other fourteen (14) youth were making
fun of. The facilitator immediately stopped the disruption and ensured the youth was ok. The
facilitator then spoke with the group about their actions and how it is not ok to call the other youth
names. Twelve (12) of the fourteen (14) youth admitted they were wrong. The other two did not
care what the facilitator said about them calling the other youth names. The group ended at
12:08p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program did have facilitator turnover during the annual review period. The facilitator
turnover did not adversely affect the delivery of Seeking Safety because the two therapeutic
managers who left were not conducting groups at the time of their departure. The primary service
is an open group held year-round. There were no gaps in service during the annual review period.
Several attendance sheets documented the use of supplemental or supportive sessions for youth
who were not present for the full hour-long group. The attendance sheet dated 5/28/2022 shows
two youth only received a fifteen (15)-minute supplemental session. The attendance sheet dated
4/19/2022 shows three youth received a fifteen (15)-minute supplemental session and there were
eleven (11) youth who received the full hour session. The attendance sheet on 4/12/2022 shows
one youth received an eight-minute supplemental session. The attendance sheet dated 3/19/2022
shows five youth received a fifteen (15)-minute supportive session. The attendance sheet dated
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
2/19/2022 shows three youth received a fifteen (15)-minute supportive session. The attendance
sheet dated 2/8/2022 shows one youth received a fifteen (15)-minute supportive session.
According to the program's Standardized Program Evaluation Protocol procedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety (90) days. The process encourages personal growth and development,
recognizes areas of improvement in an effort to reduce staff turnover. In addition, the program
provides positive incentives to improve employee motivation and dedication. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed thirteen (13) internal fidelity monitorings for facilitators of Seeking
Safety. All facilitators had fidelity checks completed during the annual review period. Four trained
staff members (clinical director and three therapeutic managers) conducted the monthly internal
fidelity checks during the annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions needed for facilitators of Seeking Safety.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event an issue identified during the primary service is detrimental to the youth’s safety and
/or treatment, the matter will be taken up directly with the program’s Human Resource Manger
and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed eight staff performance evaluations (recreation therapist, four
therapeutic managers, a senior coach counselor, the clinical director, and the compliance and
training coordinator). The evaluations do not have a section labeled to specifically address the
staff’s skill in delivering Seeking Safety. The evaluator will put comments in the narrative box
under the Direct Care Performance Factors. The evaluations do not list the specific services the
staff are trained in and facilitating. Each evaluation has similar verbiage about the staff’s delivery
of the primary services. The technical assistance specialist spoke with the clinical director and the
regional clinical director and explained each performance review should have the primary services
the facilitator is trained in listed and the evaluations should specifically address the staff’s
proficiency in delivering each primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 13 youth in the
sample, 77% (10 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 40 contact hours for this type of service. Of the 13 youth in the
sample, 38% (5 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 6 youth
High = 6 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 46%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 46% (6 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 25 25 100%
Totals 83 95 87%
This SPEP report evaluates Seeking Safety, an intervention delivered at Hope Forest Academy.
16 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 13 total youth sampled, only
12 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 10
received at least the recommended weeks of service. Youth in the sample completed between 13
and 41 weeks of service, with an average of 29 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 5 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 28 and 48 hours of
service, with an average of 39 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Hope Forest SS.pdf


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Aggression Replacement Training
SPEP Review Period: June 1, 2021-May 31, 2022
SPEP Review Conducted: June 23, 2022
Florida Department of Juvenile Justice
Report Date: 9/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2021 through May 31, 2022 shows the
program utilizes Aggression Replacement Training (ART) as a primary service. There were six
staff who were trained to facilitate the primary service: the clinical director, four therapeutic
managers, and the training and compliance coordinator. Three therapeutic managers facilitated
ART during the review period. Aggression Replacement Training is provided by a qualified trainer
over a three-day period. All trained staff can conduct the monthly internal fidelity checks. The
clinical director, compliance and training coordinator, two therapeutic managers, and the regional
clinical director conducted the monthly internal fidelity checks during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program utilizes the Aggression Replacement Training (ART) manual. The manual is
broken down into six sections. The first section covers Skillstreaming. The second section deals
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
with anger control. The third section covers moral reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's use
and include training step guides, fidelity forms and hassle logs, and the program materials. The
program does not have separate student workbooks to give out. The facilitators make copies of
the exercises before each session to pass out to the youth.
The curriculum is to be delivered to the youth over a minimum of ten (10) weeks for thirty (30)
hours. The program keeps all SPEP documents in a binder labeled "Aggression Replacement
Training" in the clinical director’s office. The program provides the service three days per week
(Tuesdays, Thursdays, and Fridays) for one hour each. ART is a closed group. All youth in the
program receive Aggression Replacement Training.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Asession of Aggression Replacement Training was observed on 6/21/2022 via Microsoft Teams.
There were eight youth present for the group along with one staff and the facilitator. The group
began at 11:05am and was held in the therapeutic manager’s office in the Administration building.
The facilitator was knowledgeable of the material and followed the lesson plan. The facilitator
ensured the youth were engaged and interactive during group. One youth appeared to be dozing
off and the facilitator immediately addressed the youth and got him engaged. The facilitator went
over all steps learned during the previous lessons. The facilitator ensured the youth understood all
the concepts by asking clarifying questions. The facilitator even challenged a few youth by asking
the question “what happens if the other person said no?” The group ended at 12:03p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program did have facilitator turnover. Two therapeutic managers left during the review
period; one left during a cohort. The program transitioned the group to a new trained facilitator.
There were five cohorts completed and one cohort ongoing at the time of the annual review. The
dates of the cohorts are: 4/6/2021-6/21/2021, 7/9/2021-9/17/2021, 9/21/2021-11/29/2021,
12/6/2021-2/25/2022, 3/8/2022-5/13/2022, and 5/24/2022-current. There was an eighteen (18)-
day gap in service between the cohort ending on 6/21/2021 and the cohort beginning on
7/9/2021.
The 4/6/2021 cohort began with eight youth and ended with seven youth. The attendance sheet
dated on 6/1/2021 shows the cohort was on lesson twenty-eight (28). The next attendance sheet
dated on 6/10/2021 shows the cohort went back to lesson twenty-three (23). The attendance
sheets dated on 6/17/2021 shows the cohort moved back to lesson twenty-nine (29) and then
lesson thirty (30) on 6/21/2021. The technical assistance specialist spoke with the clinical director
and facilitators and stated the groups needed to be run in order.
The 7/9/2021 cohort began with three youth and ended with two youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The 9/21/2021 cohort began with three youth and ended with three youth.
The 12/6/2021 cohort began with five youth and ended with six youth. One youth was added on
12/13/2021. A second youth was added on 12/16/2021. One youth was removed from the cohort
on 1/24/2022. The two youth added after the cohort began were provided with supplemental
supportive sessions to “catch them up” with the lessons they missed. Each supportive session
was documented on a separate attendance sheet and noted the supplemental sessions lasted for
an hour each.
The 3/8/2022 cohort began with seven youth and ended with six youth. The main facilitator left the
program abruptly at the end of April 2022 and the group was transitioned to another trained
facilitator beginning with the session on 5/3/2022. One youth was on transport beginning
3/13/2022 and was then dropped from the group on 4/6/2022. An attendance sheet dated
3/16/2022 listed the time of the session for 11:05a-11:35a. On 6/14/2022, the clinical director
marked through the end time and changed it to 12:05p and initialed the change. The clinical
director informed the technical assistance specialist on 6/17/2022 about the change. The clinical
director stated the progress note dated for 3/16/2022 showed the group was ran for one hour and
ended at 12:05p. The progress note was entered on 3/16/2022. The clinical director provided the
technical assistance specialist with a copy of the progress note.
The 5/24/2022 cohort began with eight youth and has the same eight youth at the end of the
review period.
According to the program's Standardized Program Evaluation Protocol procedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety (90) days. The process encourages personal growth and development,
recognizes areas of improvement in an effort to reduce staff turnover. In addition, the program
provides positive incentives to improve employee motivation and dedication. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed thirty-five (35) internal fidelity monitorings for facilitators of
Aggression Replacement Training (ART). The clinical director, compliance and training
coordinator, two therapeutic managers, and the regional clinical director conducted the monthly
internal fidelity checks during the annual review period. The technical assistance specialist
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
informed the clinical director in June 2021 there needed to be three fidelity monitorings per
facilitator per month for the primary service. There should be one fidelity check for the
Skillstreaming portion, one for the Anger Control portion, and one for the Moral Reasoning portion
of the service each month for each facilitator. Completing three fidelity checks per facilitator per
month would ensure the program would meet the recommended twenty (20) percent monitoring
for each section of the service during the cohort. The program completed the three fidelities per
month per facilitator for each month with the exception of June 2021.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions noted nor taken for facilitators of Aggression Replacement Training.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event an issue identified during the primary service is detrimental to the youth’s safety and
/or treatment, the matter will be taken up directly with the program’s Human Resource Manger
and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed eight staff performance evaluations (recreation therapist, four
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
therapeutic managers, a senior coach counselor, the clinical director, and the compliance and
training coordinator). The evaluations do not have a section labeled to specifically address the
staff’s skill in delivering Aggression Replacement Training. The evaluator will put comments in the
narrative box under the Direct Care Performance Factors. The evaluations do not list the specific
services the staff are trained in and facilitating. Each of the evaluations have similar verbiage
about the staff’s delivery of the primary services. The technical assistance specialist spoke with
the clinical director and the regional clinical director and explained each performance review
should have the primary services the facilitator is trained in listed and the evaluations should
specifically address the staff’s proficiency in delivering each primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 16 youth in the
sample, 100% (16 of 16) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 16 youth in the
sample, 100% (16 of 16) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 8 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (16 of 16) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (8 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 100 100 100%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Hope
Forest Academy.
16 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program earned 10 points for Amount of Service: Duration. Of the 16 total youth sampled, all
16 included dosage with end dates in the EBS Module. Of those youth, 16 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 15 weeks of
service, with an average of 12 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, all 16 included dosage in the EBS Module. Of those youth, 16 received at least the
recommended hours of service. Youth in the sample completed between 30 and 31 hours of
service, with an average of 30 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2021-22 Hope Forest ART.pdf

service_type: Aggression Replacement Training