summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc.
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 6/16/2020
Florida Department of Juvenile Justice
Report Date(s): 6/22/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of June 2019 through May 2020 showed there
were ten staff trained in Aggression Replacement Training (ART). There were five staff who
facilitated ART during the review period; one therapeutic manager who was running groups is no
longer employed with the program.
There were multiple attendance sheets without dates on them nor did they note the lesson being
taught. Those attendance sheets were not the proper attendance sheets. The documentation also
showed ART was not being delivered in the proper order. The attendance sheets reviewed
showed the facilitators would not follow the lesson outline. The facilitators would skip around and
not follow the order of the lessons. The new clinical director began on January 2, 2020. The
Technical Assistance Specialist discussed the issues with the attendance sheets and went over
the Standardized Program Evaluation Protocol for ART. The documentation for the cohort
beginning on 3/30/2020 shows the facilitator was filling out the attendance sheets properly and
was providing the service to the fidelity of the model.
According to the program’s Standardized Program Evaluation Protocol, the facilitator for each
primary service will have completed a formalized training for that primary service from a qualified
trainer. In order to enhance the continuity of the service due to unexpected facilitator absence and
provide more individualized instruction as needed during the service, when possible, Hope Forest
Academy will make all efforts to have a co-facilitator for the primary service. The co-facilitator will
have completed a formalized training for the primary service from a qualified trainer. For a primary
service without a formalized training, the facilitator will receive training from a licensed clinician.
This training will be documented in the facilitator’s On the Job Training Plan. The facilitator will
receive additional training on the curriculum as needed from the licensed clinician documented
within clinical supervision. All facilitators of a primary service will receive training prior to
conducting the services.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training manual. The manual is broken down
into six sections. The first section covers Skillstreaming. The second section deals with anger
control. The third section covers moral reasoning. Each of the first three sections has a script for
the facilitator to use for each lesson. Sections four through six are for the facilitator's use. It
includes training step guides, fidelity forms and hassle logs, and the program materials.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training was unable to be oberved. Documentation shows
the latest cohort ended on 6/8/2020. The program does not have enough youth to begin a new
cohort due to a low census.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program had facilitator turnover during the annual review period. The facilitator turnover did
not affect the group facilitation. The documentation shows a therapeutic manager left the
program, but the therapeutic manager finished with the cohort prior to leaving the program. The
program also gained a new clinical director on 1/2/2020.
There was a letter dated 2/27/2020 from the clinical director which stated ART was not being run
due to the lack of youth in the program. The next cohort began on 3/30/2020 and ended on
6/8/2020. There are currently only seven youth in the program at the time of the annual review.
There are no youth on the waiting list due to COVID-19.
According to the program's Standardized Program Evaluation Protocol, Hope Forest Academy
utilizes a probationary review status process designed for effective management of risk though
individualized coaching, training, and verification activities of all new employees during their first
ninety days. The process encourages personal growth and development, recognizes areas of
improvement, in an effort to reduce staff turnover. In addition, the program provides positive
incentives, which in their experience has shown to improve employee motivation and dedication,
leading to better employee retention. Through this policy, employees receive credit for years of
service, employment referrals, and for contributing to the positive culture, teamwork environment
and for going above and beyond normal job expectations. Each employee who receives credit for
doing these things is entered in a monthly drawing at each facility. In the event of facilitator
turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-facilitator is
not present, a new qualified facilitator for the primary service will take over conducting the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was no internal fidelity monitoring completed for the primary service prior to May 2020. The
regional clinical director stated the former clinical director did not complete any monitoring for
Aggression Replacement Training and she is no longer employed at the program. The Technical
Assistance Specialist informed the new clinical director she would need to be trained in the
primary service in order to complete the internal monthly fidelity monitoring. Additionally, it was
explained there needs to be one fidelity completed per section of the service per month per
facilitator, meaning there should be three instances of fidelity monitoring each month for each
facilitator.
The documentation showed there were three internal fidelity monitorings completed in May 2020.
The dates were 5/18/2020, 5/17/2020, and 5/16/2020.
According to the program's Standardized Program Evaluation Protocol, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring tool
developed specifically for that primary service by the developer of the manual or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were three fidelity
monitorings completed for Aggression Replacement Training. Those fidelity checks did not
identify any corrective actions needed.
As fidelity monitoring was not conducted regularly during the review period, no points may be
awarded in this section.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
monitoring in a note and placed in the clinical supervision file. If, during additional fidelity
monitoring, the previously identified areas continue to be an issue, the next stage is completing a
written warning and placing the facilitator on a performance improvement plan. The performance
improvement plan will state the issue identified, the tasks the facilitator needs to complete to
correct the issues, and a target date for completion. The plan will then be reviewed with the
facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor. In the
event an issue identified during the primary service is detrimental to the youth’s safety and /or
treatment, the matter will be taken up directly with the program’s Human Resource Manger and
be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 2019 through May 2020 shows
there were staff evaluations completed on the facilitators of Aggression Replacement Training.
The evaluations do not have a section to specifically address the staff’s performance in the
primary service. The Technical Assistance Specialist spoke with the clinical director and explained
the evaluations should address the staff’s proficiency in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 9 youth in the
sample, 89% (8 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 9 youth in the
sample, 100% (9 of 9) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 5 youth
High = 4 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (9 of 9) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 44% (4 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 88 100 88%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Hope
Forest Academy.
21 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Duration. Of the 9 total youth sampled, only
8 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 8
received at least the recommended weeks of service. Youth in the sample completed 13 weeks of
service.
The program earned 10 points for Amount of Service: Contact Hours. Of the 9 total youth
sampled, all 9 included dosage in the EBS Module. Of those youth, 9 received at least the
recommended hours of service. Youth in the sample completed 30 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator; by ensuring any corrective action
needed is identified, applied, and demonstrated; and by ensuring each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

report_type: SPEP

program_name_raw: Hope Forest Academy

program_status_note: Closed

service_type: Aggression Replacement Training


state: FL

facility_name: hope forest academy

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc.
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Impact of Crime
SPEP Review Date(s): 6/16/2020
Florida Department of Juvenile Justice
Report Date(s): 6/16/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of the documentation for the annual review period of June 2019-May 2020 showed there
were four facilitators trained in Impact of Crime (IOC). The documentation showed there were
three staff who facilitated IOC during the review period.
According to the program’s Standardized Program Evaluation Protocol, the facilitator for each
primary service will have completed a formalized training for that primary service from a qualified
trainer. In order to enhance the continuity of the service due to unexpected facilitator absence and
provide more individualized instruction as needed during the service, when possible, Hope Forest
Academy will make all efforts to have a co-facilitator for the primary service. The co-facilitator will
have completed a formalized training for the primary service from a qualified trainer. For a primary
service without a formalized training, the facilitator will receive training from a licensed clinician.
This training will be documented in the facilitator’s On the Job Training Plan. The facilitator will
receive additional training on the curriculum as needed from the licensed clinician documented
within clinical supervision. All facilitators of a primary service will receive training prior to
conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime Curriculum. Then the manual goes over all seven chapters
of the group and provides a script for the facilitator to follow. The manual also has an appendix
and a student guide with all the student exercises.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed. The most recent cohort ended on
5/20/2020. The program does not have enough youth to begin a new cohort due to a low census.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program had facilitator turnover during the review period which adversely affected facilitation
of Impact of Crime groups. One facilitator began a group in December 2019 and abruptly left the
program in January 2020. The cohort was discontinued and only one youth in the group was able
to receive IOC in another cohort. The former training coordinator and recreation therapist also left
the program. The recreation therapist left the program in August 2019. There was a gap in service
from the end of August 2019 through 10/21/2019. Another trained facilitator picked up where the
recreation therapist left off instead of starting from the beginning. The recreation therapist
returned to the program and began facilitating groups. The recreation therapist left the program
again at the end of the cohort in May 2020. A review of documentation for the annual review
period of June 2019 through May 2020 shows there were three cohorts completed.
There were multiple attendance sheets which noted the lesson but were otherwise filled out
incorrectly. The Technical Assistance Specialist reviewed the attendance sheets with the new
clinical director and discussed the proper way to document the primary service groups.
According to the program's Standardized Program Evaluation Protocol, Hope Forest Academy
utilizes a probationary review status process designed for effective management of risk though
individualized coaching, training, and verification activities of all new employees during their first
ninety days. The process encourages personal growth and development, recognizes areas of
improvement, in an effort to reduce staff turnover. In addition, the program provides positive
incentives, which in their experience has shown to improve employee motivation and dedication
and thus leads to better employee retention. Through this policy, employees receive credit for
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation for the annual review period shows there were three internal fidelity
monitorings completed for the primary service. Fidelity monitoring was conducted on 11/14/2019,
4/16/2020, and 5/2/2020. The two fidelity checks completed by the training coordinator were not
filled out correctly. The training coordinator did not fill out the top of the form which includes the
date and time of the session, the facilitator conducting the session, and the observer conducting
the fidelity check. The reviewer had to cross reference the date the form was signed with the
attendance sheet to determine the facilitator.
The Technical Assistance Specialist informed the new clinical director she would need to be
trained in Impact of Crime in order to complete the internal monthly fidelity monitoring or she could
have a trained facilitator conduct the fidelity monitoring.
According to the program's Standardized Program Evaluation Protocol, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring tool
developed specifically for that primary service by the developer of the manual or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed shows there were three fidelity monitorings completed for Impact of
Crime. Those fidelity checks did not identify any corrective actions needed.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If, during additional fidelity
monitoring, the previously identified areas continue to be an issue, the next stage is completing a
written warning and placing the facilitator on a performance improvement plan. The performance
improvement plan will state the issue identified, the tasks the facilitator needs to complete to
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
correct the issues, and a target date for completion. The plan will then be reviewed with the
facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor. In the
event an issue identified during the primary service is detrimental to the youth’s safety and /or
treatment, the matter will be taken up directly with the program’s Human Resource Manger and
be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 2019 through May 2020 shows
there were staff evaluations completed on the facilitators of Impact of Crime (IOC). The
evaluations do not have a section to specifically address the staff’s performance in delivering IOC.
There is one staff’s evaluation which addresses his performance delivering IOC. It is addressed in
the performance factors comment section. The Technical Assistance Specialist spoke with the
clinical director and explained the evaluations should address the staff’s proficiency in delivering
the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 20 youth in the
sample, 5% (1 of 20) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 22 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 88%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 11 youth
High = 6 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 6 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (19 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 30% (6 of 20) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 22 25 88%
Totals 52 85 61%
This SPEP report evaluates Impact of Crime, an intervention delivered at Hope Forest Academy.
21 youth completed the program during the review period. Of those youth, 21 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 21 total youth sampled, only
1 included dosage with end dates in the EBS Module. That youth did not receive at least the
recommended weeks of service, completing 6 weeks of service.
The program earned 0 points for Amount of Service: Contact Hours. Of the 21 total youth
sampled, only 1 included dosage in the EBS Module. That youth received at least the
recommended hours of service, completing 24 hours of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 22 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups; by ensuring fidelity monitoring is completed every
month for each facilitator; and by ensuring each facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

service_type: Impact of Crime


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc.
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Seeking Safety
SPEP Review Date(s): 6/16/2020
Florida Department of Juvenile Justice
Report Date(s): 6/22/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of the documentation for the annual review period of June 2019 through May 2020
showed there were nine staff who were trained in Seeking Safety. The training is conducted by
the clinical director as an in-service training. All staff who facilitated the primary service were
trained prior to conducting their first group.
According to the program’s Standardized Program Evaluation Protocol, the facilitator for each
primary service will have completed a formalized training for that primary service from a qualified
trainer. In order to enhance the continuity of the service due to unexpected facilitator absence and
provide more individualized instruction as needed during the service, when possible, Hope Forest
Academy will make all efforts to have a co-facilitator for the primary service. The co-facilitator will
have completed a formalized training for the primary service from a qualified trainer. For a primary
service without a formalized training, the facilitator will receive training from a licensed clinician.
This training will be documented in the facilitator’s On the Job Training Plan. The facilitator will
receive additional training on the curriculum as needed from the licensed clinician documented
within clinical supervision. All facilitators of a primary service will receive training prior to
conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance Abuse for
their primary service. The manual is broken down into twenty-five sessions. Each session
explains the topic the therapist will cover and provides handouts for the youth to complete. The
primary service can be utilized for individual treatment or group treatment.
Pursuant to the facility's Standard Program Evaluation Protocol, each primary service will be
facilitated in accordance to its manual. The primary service will be conducted as indicated in the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
manual/protocol; this includes adherence to the session plan, group size, population, frequency,
needed materials, duration, and interventions. The facilitator of the primary service will specify the
topic covered, interventions utilized, duration of the service, youth present, and progress for each
session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 6/16/2020. The facilitator began group at 10:58a.
The facilitator began with having the youth state the motto and group norms. The youth were
asked to check in and were asked to state how they were doing by relating their feeling to an
emoji. The majority of the seven youth did not want to answer the questions or participate in
group. The youth were not volunteering to answer the questions. The facilitator tried to get the
youth to answer the questions. The facilitator would randomly choose a youth and try to get them
involved. The youth began answering questions about half-way through the group. The facilitator
asked the youth who their most supportive, neutral, and destructive people were in their life. The
facilitator then went over what each type of person can do in your life. The facilitator discussed
what the youth can do to help tell others how to support them in their recovery. The facilitator
asked the youth about their recovery progress at Hope Forest. The group ended with their closing
pledge at 11:58a.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program had facilitator turnover during the annual review period; however, the turnover did
not adversely affect facilitation of Seeking Safety. The documentation showed the therapeutic
manager who left the program was not running groups at the time of her departure. The program
hired a new clinical director who began on 1/2/2020. Seeking Safety is an open group and is run
year-round.
According to the program's Standardized Program Evaluation Protocol, Hope Forest Academy
utilizes a probationary review status process designed for effective management of risk though
individualized coaching, training, and verification activities of all new employees during their first
ninety days. The process encourages personal growth and development, recognizes areas of
improvement, in an effort to reduce staff turnover. In addition, the program provides positive
incentives, which in their experience has shown to improve employee motivation and dedication
and thus leads to better employee retention. Through this policy, employees receive credit for
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was no internal fidelity monitoring completed for Seeking Safety prior to April 2020. The
regional clinical director stated the former clinical director did not complete any monitoring for
Seeking Safety. The Technical Assistance Specialist informed the new clinical director she would
need to be trained in Seeking Safety in order to complete the internal monthly fidelity monitoring.
The documentation showed there were two internal fidelity monitorings completed during the
annual review period. The dates were 4/8/2020 and 5/6/2020.
According to the program's Standardized Program Evaluation Protocol, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring tool
developed specifically for that primary service by the developer of the manual or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were two fidelity
monitoring completed for the primary service. Those fidelity checks did not identify any corrective
actions needed.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If, during additional fidelity
monitoring, the previously identified areas continue to be an issue, the next stage is completing a
written warning and placing the facilitator on a performance improvement plan. The performance
improvement plan will state the issue identified, the tasks the facilitator needs to complete to
correct the issues, and a target date for completion. The plan will then be reviewed with the
facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor. In the
event an issue identified during the primary service is detrimental to the youth’s safety and /or
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
treatment, the matter will be taken up directly with the program’s Human Resource Manger and
be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed showed there were staff evaluations completed on the facilitators of
Seeking Safety. The evaluations do not have a section to specifically address the staff’s
performance in the primary service. The Technical Assistance Specialist spoke with the clinical
director and explained the evaluations should address the staff’s proficiency in delivering the
primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 1 youth in the
sample, 100% (1 of 1) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 1 youth in the
sample, 0% (0 of 1) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 1 youth
Total Youth in Sample = 1 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 100%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 1 youth
Total Youth in Sample = 1 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (1 of 1) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 100% (1 of 1) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 85 95 89%
This SPEP report evaluates Seeking Safety, an intervention delivered at Hope Forest Academy.
21 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The dosage information included in this report is for one youth only; this small sample size is not
necessarily an accurate reflection of the effectiveness of this service at Hope Forest Academy.
The program earned 10 points for Amount of Service: Duration. The 1 youth sampled included
dosage with end dates in the EBS Module. That youth received at least the recommended weeks
of service, completing 26 weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. The 1 youth sampled
included dosage in the EBS Module. That youth did not receive at least the recommended hours
of service, completing 25 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and by ensuring each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Seeking Safety


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Hope Forest Academy
Rite of Passage, Inc
(Contract Provider)
46821 Jane Lane
Paisley, Florida 32767
Primary Service: Impact of Crime
SPEP Review Period: 6/1/2020-5/31/2021
SPEP Review Conducted: 6/16/2021-6/17/2021
Florida Department of Juvenile Justice
Report Date: 8/26/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation for the annual review period of June 1, 2020 through May 31, 2021 shows the
program utilizes Impact of Crime (IOC) as their primary service. There are four staff who are
trained to facilitate the primary service. Two of the four trained staff facilitated the primary service
during the review period. The trained staff are one therapeutic manager, two shift supervisors,
and one coach counselor. The training is a twenty-eight (28) hour in-person training, conducted
over three days by a qualified trainer. All trained staff members can conduct the internal fidelity
checks. Three trained staff members conducted the monthly internal fidelity checks during the
annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Hope Forest Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens
with an introduction to the Impact of Crime Curriculum. Then the manual goes over all seven
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
chapters of the manual and provides a script for the facilitator to follow. The manual also has an
appendix and a student guide with all the student exercises. The program does not have separate
student workbooks to give out. The facilitators make copies of the exercises before each session
to pass out to the youth.
The group is a closed group with no more than twelve youth in each cohort. The curriculum is to
be delivered to the youth over a minimum of sixteen weeks for thirty-two sessions. The program
keeps all SPEP documents in a binder labeled Impact of Crime. The program provides the service
two times per week for one hour each. All youth receive Impact of Crime.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 6/17/2021. There were eight youth present for the
group along with one staff and the facilitator. The group began at 12:10pm and was held in the
classroom in the Administration building. The facilitator was knowledgeable of the material and
followed the lesson plan. The youth were provided lunch during the group. The youth were
arguing with the coach counselors about their lunch. There were several youth who were not
paying attention to the documentary being shown during group. The facilitator debriefed the youth
after the documentary was over. The group ended at 1:08p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program did not have any facilitator turnover during the annual review period. The
program also gained a new clinical director on 5/1/2021. There were two cohorts completed
during the annual review period. The dates of the cohorts are: 4/8/2021-current and 11/1/2020 -
5/11/2021. There were no attendance sheets provided for the months of June-October 2020. The
new clinical director stated the program did not run the primary service during that time because
there were no new youth admitted to the program needing the primary service; however, two of
the four youth missing dosage for this service were at the program during this time.
The 11/1/2020 cohort began with eight youth and ended with four youth. One youth was provided
supplemental sessions to ensure the youth finished the primary service prior to release from the
program.
The 4/8/2021 cohort began with ten youth and currently has eight youth. The cohort began with
the therapeutic manager facilitating the cohort. A shift supervisor took over facilitating groups on
4/21/2021. On 4/27/2021, two youth were given supplemental sessions due to leaving the
program before their cohort would finish the service. The clinical director stated the therapeutic
manager was conducting the supplemental sessions for the two youth.
According to the program's Standardized Program Evaluation Protocol procedure, Hope Forest
Academy utilizes a probationary review status process designed for effective management of risk
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, in an effort to reduce staff turnover. In addition, the program provides
positive incentives, which in their experience has shown to improve employee motivation and
dedication and leads to better employee retention. Through this policy, employees receive credit
for years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed eleven (11) internal fidelity monitorings of Impact of Crime. The
program did not conduct internal fidelity checks in June 2020, July 2020, August 2020, September
2020, or October 2020 because there were no cohorts being run during that time.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period shows there were no identified
corrective actions needed for facilitators of Impact of Crime.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manger and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of June 1, 2020 through May 31, 2021
shows the program completed three staff evaluations. The evaluations do not have a section to
specifically address the staff’s performance in facilitating Impact of Crime. The technical
assistance specialist spoke with the clinical director and the facility administrator and explained
each performance review should have the primary services the facilitator is trained in listed and
the evaluations should specifically address the staff’s proficiency in delivering each primary
service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 13 youth in the
sample, 31% (4 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 13 youth in the
sample, 38% (5 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 100 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 54%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 54% (7 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 25 25 100%
Totals 59 85 69%
This SPEP report evaluates Impact of Crime, an intervention delivered at Hope Forest Academy.
16 youth completed the program during the review period. Of those youth, 13 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 13 total youth sampled, only
9 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 4
received at least the recommended weeks of service. Youth in the sample completed between 13
and 27 weeks of service, with an average of 17 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, only 9 included dosage in the EBS Module. Of those youth with correct dosage, 5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
received at least the recommended hours of service. Youth in the sample completed between 11
and 31 hours of service, with an average of 20 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Hope Forest Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, there are no unnecessary gaps in service, and each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Hope Forest Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2020-21 Hope Forest IOC.pdf