summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Impact of Crime
SPEP Review Period: June 1, 2021 - May 31, 2022
SPEP Review Conducted: June 23, 2022
Florida Department of Juvenile Justice
Report Date: 9/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2021 through May 31, 2022 shows the
program utilizes Impact of Crime (IOC) as their primary service. There were six staff trained to
facilitate IOC: one training and compliance coordinator, the clinical director, a coach counselor,
recreation therapist, and two shift supervisors. The program had one other trained facilitator (shift
supervisor) from another program assist delivering the primary service. Three of the six trained
program staff (clinical director, compliance and training coordinator, and a shift supervisor) were
listed as the main facilitator for their cohorts during the review period. Two additional trained
program staff (two shift supervisors) and the one trained shift supervisor from another program
facilitated a few groups during the review period. IOC training is an in-person training which is
twenty-eight (28) hours long, conducted over three days by a qualified trainer. All trained staff
members can conduct the internal fidelity checks. Three trained staff members conducted the
monthly internal fidelity checks during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program utilizes the Impact of Crime: Addressing the Harm v 4.0 manual. The manual
opens with an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all
seven chapters of the manual and provides a script for the facilitator to follow. The manual also
has an appendix and a student guide with all the student exercises. The program does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth.
IOC is delivered as a closed group with no more than twelve (12) youth in each cohort. The
curriculum is to be delivered to the youth over a minimum of sixteen (16) weeks for thirty-two (32)
sessions. The program keeps all SPEP documents in a binder labeled "Impact of Crime" in the
clinical director’s office. The program provides the service Tuesdays and Thursdays for one hour
each. All youth receive Impact of Crime.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 6/23/2022 via Microsoft Teams. There were ten
(10) youth and the facilitator present. The group was held in the therapeutic manager’s office in
the Administration building and began at 12:00p. The facilitator was knowledgeable of the material
and followed the lesson plan. The facilitator was able to redirect the youth when they were getting
off topic. The facilitator engaged the youth and asked clarifying questions to ensure the youth
understood the concepts. The youth started to become disinterested in the material towards the
end of the group. The youth told the facilitator they were tired and did not like they had to go to
three groups each day. Some of the youth were trying to get behind the case manager’s desk, but
the facilitator immediately informed the youth they were to stay in front of the desk. The group
ended at 1:03p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program did have facilitator turnover during the annual review period. One facilitator (a
shift supervisor) went on paternity leave and then transferred to another program shortly after
coming back. The shift supervisor was facilitating a cohort at the time of their departure. Another
shift supervisor who left during the annual review period was not actively facilitating any cohorts.
There were four Impact of Crime cohorts completed during the annual review period. The dates of
the cohorts were: 4/8/2021-8/6/2021, 4/8/2021-6/22/2021, 7/2/2021-12/8/2021 and 12/3/2021-
5/18/2022.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The 4/8/2021 cohort began with ten (10) youth and ended with six youth. The compliance and
training coordinator took over facilitating the cohort for the shift supervisor on 6/9/2021.
The second cohort which began on 4/8/2021 began with two youth and ended with two youth. The
last sign in sheet was dated for 6/22/2021 and shows the cohort stopped on chapter six objective
three. There were no other attendance sheets provided because the youth were released from the
program prior to completing the primary service.
The 7/2/2021 cohort began with two youth and ended with one youth. The attendance sheets
show the youth were on chapter one objective one on 7/2/2021 and then on chapter six objective
four on 7/7/2021. On 7/9/2021, the facilitator went back to chapter one objective two. On
8/4/2021, the youth were working on chapter seven objectives four and five. On 8/9/2021, the
facilitator went back to chapter two objective two. The clinical director and compliance and training
coordinator stated they were trying to get the youth through the service because a youth was
scheduled to leave prior to the cohort’s projected completion date. The technical assistance
specialist informed the clinical director and the compliance and training coordinator the primary
service is supposed to be delivered in order. The topics build on each other and the youth lose out
if they are not delivered properly. One youth was discharged from the program on 11/19/2021.
The 12/3/2021 cohort began with seven youth and ended with six youth. Three youth were added
to the cohort on 12/22/2021. The clinical director stated the three youth entered the program right
after the start of the cohort. The youth were placed in the cohort and received supplemental
sessions to ensure they were caught up, in accordance with the Department's guidance. One
additional youth appeared on the 12/29/2021 attendance sheet which would make eleven youth in
the group. The clinical director stated the additional youth who signed the attendance sheet on
12/29/2021 was not assigned to this cohort. The youth did not attend any subsequent groups for
this cohort once staff realized he was not supposed to be in the current cohort. The clinical
director stated the youth is set to be in the next cohort. One youth left the group on 1/5/2022. The
note on the attendance sheet stated the youth decided he did not want to be a part of the group
since he is leaving the program. The clinical director stated the youth was transferred to another
facility and not released. The youth also refused to attend two of the previous sessions. Another
youth was released from the program on 1/10/2022 prior to completing the primary service. One
youth was released on 3/17/2022. The recreation therapist took over facilitating the cohort on
4/4/2022. One youth was released from the program on 4/28/2022.
According to the program's Standardized Program Evaluation Protocol procedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety (90) days. The process encourages personal growth and development,
recognizes areas of improvement in an effort to reduce staff turnover. In addition, the program
provides positive incentives to improve employee motivation and dedication. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed twelve (12) internal fidelity monitorings for facilitators of Impact of
Crime. There was a note dated 6/30/2021 in the SPEP binder which stated an internal fidelity
monitoring was not completed on the shift supervisor due to paternity leave and being transferred
to another facility. The shift supervisor facilitated two sessions in June on 6/3/2021 and 6/5/2021.
The program was missing one internal fidelity monitoring for the compliance and training
coordinator for the month of December 2021. The clinical director stated the compliance and
training coordinator only facilitated one group during the month of December 2021, so a fidelity
monitoring was not completed. The compliance and training coordinator, a shift supervisor, and
the clinical director conducted the monthly internal fidelity monitorings.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions needed for facilitators of Impact of Crime.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event an issue identified during the primary service is detrimental to the youth’s safety and
/or treatment, the matter will be taken up directly with the program’s Human Resource Manger
and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed eight staff performance evaluations (recreation therapist, four
therapeutic managers, a senior coach counselor, the clinical director, and the compliance and
training coordinator). The evaluations do not have a section labeled to specifically address the
staff’s skill in delivering Impact of Crime. The evaluator will put comments in the narrative box
under the Direct Care Performance Factors. The evaluations do not list the specific services the
staff are trained in and facilitating. Each evaluation has similar verbiage about the staff’s delivery
of the primary services. The technical assistance specialist spoke with the clinical director and the
regional clinical director and explained each performance review should have the primary services
the facilitator is trained in listed and the evaluations should specifically address the staff’s
proficiency in delivering each primary service.
The program can earn 1 point if facilitator’s performance evaluation includes discussion of their
skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 16 youth in the
sample, 75% (12 of 16) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 16 youth in the
sample, 88% (14 of 16) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 8 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (16 of 16) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (8 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 68 85 81%
This SPEP report evaluates Impact of Crime, an intervention delivered at Hope Forest Academy.
16 youth completed the program during the review period. Of those youth, all 16 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 16 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 12
received at least the recommended weeks of service. Youth in the sample completed between 6
and 28 weeks of service, with an average of 19 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, all 16 included dosage in the EBS Module. Of those youth, 14 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
recommended hours of service. Youth in the sample completed between 9 and 31 hours of
service, with an average of 28 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, fidelity monitoring is completed every month for each
facilitator, and facilitator’s performance evaluation includes discussion of their skill in delivering
this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2021-22 Hope Forest IOC.pdf

service_type: Impact of Crime


state: FL

facility_name: hope forest academy

summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Aggression Replacement Training
SPEP Review Period: June 1, 2022-May 31, 2023
SPEP Review Conducted: June 14, 2023
Florida Department of Juvenile Justice
Report Date: 10/20/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2022 through May 31, 2023 shows the
program utilizes Aggression Replacement Training (ART) as a primary service. There were four
staff who were trained to facilitate the primary service: two therapeutic managers, the clinical
director, and the director of student services. One therapeutic manager facilitated the primary
service during the annual review period. Facilitator training for ART is provided by a qualified
trainer over a three-day period. All trained staff can conduct the monthly fidelity checks. The
clinical director, the director of student services, and one therapeutic manager conducted the
internal fidelity monitoring during the annual review period. The director of clinical services and
another program’s clinical director also assisted the program by conducting monthly internal
fidelity checks during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service. For a primary service
without a formalized training, the facilitator will receive training from a licensed clinician. This
training will be documented in the facilitator’s On the Job Training Plan. All facilitators of a primary
service will receive training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program utilizes the Aggression Replacement Training (ART) manual. The manual is
broken down into six sections. The first section covers Skillstreaming. The second section covers
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Anger Control. The third section covers Moral Reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's
use. It includes training step guides, fidelity forms, hassle logs, and the program materials. The
program does not have separate student workbooks to give out. The facilitators make copies of
the exercises before each session to hand out to the youth.
The curriculum is to be delivered to the youth over a minimum of ten (10) weeks for thirty (30)
hours. The program keeps all SPEP documents in a binder labeled "Aggression Replacement
Training" in the clinical director’s office. The program provides the service three days per week
(Tuesdays, Thursdays, and Fridays) for one hour each. ART is a closed group. All youth in the
program receive ART.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training (ART) was unable to be observed. The program
only has two youth who need the service. All other youth in the program have completed the
service. ART requires a minimum of six youth to begin a new cohort. The last cohort ended on
4/21/2023. This indicator is rated as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for Aggression Replacement Training (ART) for the annual review
period of June 1, 2022 through May 31, 2023 shows the program did not have facilitator turnover.
There were four cohorts completed during the annual review period.
The first cohort began on 5/24/2022 with eight youth and ended on 7/29/2022 with seven youth.
One youth was transferred on 6/13/2022.
The second cohort began on 8/23/2022 with three youth and ended on 11/4/2022 with five youth.
One youth was added on 8/30/2022 and another youth was added on 9/2/2022. The youth were
provided with make-up sessions. There was a note stating no ART groups were ran 9/25/2022-
10/1/2022 due to the mandatory evacuation for hurricane Ian.
The third cohort began on 11/15/2022 with ten (10) youth and ended on 1/27/2023 with seven
youth. One youth transferred on 12/2/2022. Another youth transferred on 12/19/2022. A third
youth transferred on 1/6/2023. There were five attendance sheets (2/2/2023, 2/7/2023, 2/10/2023,
2/17/2023, and 2/18/2023) which documented three youth receiving make-up sessions after the
final group session.
The fourth cohort began on 2/7/2023 began with four youth and ended on 4/21/2023 with four
youth. The attendance sheet dated 2/14/2023 had no group times listed. The attendance sheet
dated 4/4/2023 had a note which stated one youth slept through the group. There was a make-up
session provided for the youth on 4/21/2023.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program completed thirty-six (36) internal fidelity monitorings for facilitators of
Aggression Replacement Training. The clinical director, the director of student services, and one
therapeutic manager conducted the internal fidelity monitoring. The director of clinical services
and another program’s clinical director also assisted the program by conducting monthly internal
fidelity checks during the annual review period. The program completed all required monthly
fidelity checks, without exception.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for Aggression Replacement Training (ART) for the annual review
period shows there were no identified corrective actions needed.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for Aggression Replacement Training (ART) for the annual review
period of June 1, 2022 through May 31, 2023 shows the program completed two staff
performance evaluations (Recreation Therapist and Therapeutic Manager). The clinical director
stated the evaluations for herself and the transition services coordinator are not posted yet so she
is unable to provide the evaluation for review. The evaluations do not have a section labeled to
address the staff’s performance in the primary service. The evaluator will put comments in each of
the narrative boxes. The evaluations provided do not list the specific services the staff are trained
in and facilitating nor do they address the facilitator’s skill in delivering the primary service.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 26 youth in the
sample, 100% (26 of 26) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 26 youth in the
sample, 100% (26 of 26) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 16 youth
High = 6 youth
Total Youth in Sample = 26 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 23%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 6 youth
Total Youth in Sample = 26 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (24 of 26) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 23% (6 of 26) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 15 25 60%
Totals 90 100 90%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Hope
Forest Academy.
26 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Duration. Of the 26 total youth sampled, all
26 included dosage with end dates in the EBS Module. Of those youth, 26 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 21 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 26 total youth
sampled, all 26 included dosage in the EBS Module. Of those youth, 26 received at least the
recommended hours of service. Youth in the sample completed between 30 and 60 hours of
service, with an average of 31 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

file_name: FY2022-23 HFYA ART.pdf

program_name_raw: Hope Forest Academy

program_status_note: Closed

service_type: Aggression Replacement Training


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Impact of Crime
SPEP Review Period: June 1, 2022-May 31, 2023
SPEP Review Conducted: June 14, 2023
Florida Department of Juvenile Justice
Report Date: 10/20/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2022 through May 31, 2023 shows the
program utilizes Impact of Crime (IOC) as their primary service. There were eight staff trained to
facilitate the primary service- the director of student services, the clinical director, a coach
counselor, a recreation therapist, two shift supervisors, a case manager, and the transition
services coordinator. Three of the eight trained staff were listed as the main facilitator for their
cohort during the annual review period- director of student services, a shift supervisor, and a case
manager. Two additional trained program staff (recreation therapist and clinical director) facilitated
a few groups during the annual review period. The training is currently held as a virtual training via
Microsoft Teams which is twenty-eight (28) hours long, conducted over three days by a qualified
trainer. All trained staff members can conduct the internal fidelity checks. Two of the eight trained
staff members (clinical director and recreation therapist) conducted the monthly internal fidelity
checks.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service. For a primary service
without a formalized training, the facilitator will receive training from a licensed clinician. This
training will be documented in the facilitator’s On the Job Training Plan. All facilitators of a primary
service will receive training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program utilizes the Impact of Crime: Addressing the Harm v 4.0 manual. The manual
opens with an introduction to the Impact of Crime curriculum. The manual then goes over all
seven chapters and provides a script for the facilitator to follow. The manual also has an appendix
and a student guide with all the student exercises. The program does not have separate student
workbooks to give out. The facilitators make copies of the exercises before each session to hand
out to the youth.
Impact of Crime (IOC) is a closed group with no more than twelve (12) youth in each cohort. The
curriculum is to be delivered to the youth over a minimum of sixteen (16) weeks for thirty-two (32)
sessions. The program keeps all SPEP documents in a binder labeled "Impact of Crime" in the
clinical director’s office. The program provides the service Tuesdays and Thursdays for one hour
each. All youth in the program receive IOC.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime (IOC) was observed on 1/31/2023. The group began at 12:05p.
There were seven youth, three staff, and the facilitator present for the group. The facilitator stated
the group would watch a video about texting and driving. After the video, the group would discuss
making good and bad decisions. The youth were provided with their lunches during the session.
The clinical director stated that lunch was the only time available for the youth to receive the
primary service. The video lasted for forty-five (45) minutes. The facilitator attempted to debrief
the youth, but the youth were losing focus and were ready to end the group. The group ended at
1:03p.
The Technical Assistance Specialist spoke with the facilitator and stated the video went a little too
long for the group. The videos should not last an entire session. The clinical director also informed
the facilitator to break up the video. The Technical Assistance Specialist advised the facilitator to
break the video up in segments and then debrief the youth after each of those shorter video
segments.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program did have facilitator turnover. The turnover negatively affected the delivery of
Impact of Crime. A case manager left the program in December 2022 and a shift supervisor left
the program in February 2023. Both were facilitating a cohort at the time they left the program.
There were four cohorts completed during the annual review period. The first cohort began on
6/21/2022 with ten (10) youth and ended on 11/1/2022 with five youth. One youth was discharged
from the program on 7/21/2022. Another youth was discharged on 7/28/2022. A third youth was
discharged on 8/16/2022. A fourth youth was arrested on 8/4/2022. A fifth youth was discharged
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
on 10/12/2022. There was a note stating no IOC groups were run 9/25/2022-10/1/2022 due to a
mandatory evacuation for Hurricane Ian.
The second cohort began on 11/17/2022 with ten (10) youth and ended on 4/11/2023 with six
youth. One youth was transferred on 12/1/2022. The recreation therapist facilitated a group on
1/5/2023 since the case manager left the program. Two youth were transferred on 1/6/2023. The
director of student services took over facilitating the cohort on 1/10/2023.
The third cohort began on 1/12/2023 with four youth and ended on 5/11/2023 with four youth. The
clinical director began the group and ran two sessions to ensure the youth were able to receive
the entire service prior to being discharged from the program. Two youth would not have received
the entire service if the program waited until they had six youth to begin the cohort. The shift
supervisor took over the cohort on 1/18/2023. The shift supervisor typed the group times on the
attendance sheets and did not hand write them. The clinical director stated she discussed this
with the shift supervisor. There was a gap in service of seventeen (17) days from 2/10/2023-
2/27/2023. There was a note stating there was a change in facilitator due to the Shift Supervisor
leaving the program to return to another program. The clinical director facilitated the group on
2/28/2023 and 3/2/2023. The transition services coordinator took over facilitating the group on
3/7/2023.
The fourth cohort began on 5/9/2023 with five youth and was ongoing with two youth at the end of
the review period. There were three youth arrested on 5/20/2023.
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program completed seventeen (17) internal fidelity monitorings for facilitators of Impact
of Crime. There were no missing internal fidelity checklists for the annual review period. The
clinical director and recreation therapist conducted the monthly internal fidelity checks during the
annual review period. The internal fidelity check completed on 7/28/2022 did not have the
facilitator listed. The attendance sheet dated 7/28/2022 shows the case manager facilitated a
group on that specific day.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for Impact of Crime (IOC) for the annual review period shows there
were no identified corrective actions needed.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program completed two staff performance evaluations (recreation therapist and
therapeutic manager). The clinical director stated the evaluations for herself and the transition
services coordinator are not posted yet so she is unable to provide the evaluation for review. The
evaluations do not have a section labeled to specifically address the staff’s performance in the
primary service. The evaluator will put comments in each of the narrative boxes. The evaluations
provided do not list the specific services the staff are trained in and facilitating nor do they address
the facilitator’s skill in delivering the primary service.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 26 youth in the
sample, 69% (18 of 26) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 26 youth in the
sample, 65% (17 of 26) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 16 youth
High = 6 youth
Total Youth in Sample = 26 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 23%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 6 youth
Total Youth in Sample = 26 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (24 of 26) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 23% (6 of 26) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 15 25 60%
Totals 57 85 67%
This SPEP report evaluates Impact of Crime, an intervention delivered at Hope Forest Academy.
26 youth completed the program during the review period. Of those youth, all 26 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 26 total youth sampled, only
21 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 18
received at least the recommended weeks of service. Youth in the sample completed between 5
and 24 weeks of service, with an average of 19 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 26 total youth
sampled, only 21 included dosage in the EBS Module. Of those youth with correct dosage, 17
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
received at least the recommended hours of service. Youth in the sample completed between 10
and 37 hours of service, with an average of 29 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 79%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, there are no unnecessary gaps in service, and each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 HFYA IOC.pdf


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Seeking Safety
SPEP Review Period: June 1, 2022-May 31, 2023
SPEP Review Conducted: June 14, 2023
Florida Department of Juvenile Justice
Report Date: 10/20/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 1, 2022 through May 31, 2023 shows the
program utilizes Seeking Safety as their primary service. There were three staff who were trained
to facilitate the primary service- the clinical director and two therapeutic managers. All three
trained staff facilitated the primary service during the annual review period. The training is
conducted by the clinical director or the director of clinical services as an in-service training. All
trained staff can conduct the monthly internal fidelity checks. Two trained program staff members
(clinical director and one therapeutic manager) as well as the director of clinical services
conducted the monthly internal fidelity checks during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service. For a primary service
without a formalized training, the facilitator will receive training from a licensed clinician. This
training will be documented in the facilitator’s On the Job Training Plan. All facilitators of a primary
service will receive training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance
Abuse for their primary service. The manual is broken down into twenty-five (25) sessions. Each
session explains the topic the therapist will cover and contains handouts for the youth to
complete. The primary service can be utilized for individual treatment or group treatment. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
program does not have separate student workbooks to give out. The facilitators make copies of
the exercises before each session to hand out to the youth.
The program holds the primary service on Wednesdays and Saturdays for one hour each session.
The program keeps their documentation for the primary service in a SPEP binder labeled Seeking
Safety. The binder contains all attendance sheets, internal fidelity monitoring checklists, and
group schedules. The training certificates are held by the training coordinator at the program. All
youth who are admitted to the program receive the primary service. The primary service is held as
an open group ran year-round.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 6/14/2023 via Microsoft Teams. There were eight
youth present for the group along with two coach counselors and the facilitator. The group began
at 11:13a. The facilitator was knowledgeable of the material and followed the lesson plan. The
facilitator ensured the youth understood the material prior to moving onto a different subject. The
group ended at 11:50a.
The Technical Assistance Specialist informed the therapeutic manager and the clinical director
that the group needs to run longer than thirty-five (35) minutes. The therapeutic manager and
clinical director acknowledged groups should run for an hour. The therapeutic manager stated she
was having difficulty accessing her email to find the Teams link.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2022 through May 31, 2023
shows the program did not have facilitator turnover during the annual review period. There were
no gaps in service during the annual review period. Seeking Safety is conducted as an open
service with groups held consistently throughout the year.
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2021 through May 31, 2022
shows the program completed fourteen (14) internal fidelity monitorings for facilitators of Seeking
Safety. All required internal fidelity checks were completed for each facilitator for the annual
review period. Two trained program staff members (clinical director and one therapeutic manager)
as well as the director of clinical services conducted the monthly internal fidelity checks during the
annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for Seeking Safety for the annual review period shows there were
no identified corrective actions needed.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for Seeking Safety for the annual review period of June 1, 2022
through May 31, 2023 shows the program completed two staff performance evaluations
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
(Recreation Therapist and Therapeutic Manager). The Clinical Director stated the evaluations for
herself and the Transition Services Coordinator are not posted yet so she is unable to provide the
evaluation for review. The evaluations do not have a section labeled to specifically address the
staff’s performance in the primary service. The evaluator will put comments in each of the
narrative boxes. The evaluations provided do not list the specific services the staff are trained in
and facilitating nor do they address the facilitator’s skill in delivering the primary service.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 21 youth in the
sample, 100% (21 of 21) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 21 youth in the
sample, 100% (21 of 21) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 15 youth
High = 4 youth
Total Youth in Sample = 21 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 19%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 4 youth
Total Youth in Sample = 21 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (21 of 21) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 19% (4 of 21) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 15 25 60%
Totals 85 95 89%
This SPEP report evaluates Seeking Safety, an intervention delivered at Hope Forest Academy.
26 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 21 total youth sampled, all
21 included dosage with end dates in the EBS Module. Of those youth, 21 received at least the
recommended weeks of service. Youth in the sample completed between 26 and 67 weeks of
service, with an average of 34 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 21 total youth
sampled, all 21 included dosage in the EBS Module. Of those youth, 21 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 40 and 118 hours of
service, with an average of 55 hours.
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring dosage for
all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 HFYA SS.pdf

service_type: Seeking Safety