summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Seeking Safety
SPEP Review Period: April 1, 2021-March 31, 2022
SPEP Review Conducted: May 4, 2022
Florida Department of Juvenile Justice
Report Date: 7/27/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2021, through March 31, 2022, shows
the program utilizes Seeking Safety as their primary service. Seeking Safety training is conducted
by the clinical director as an in-service training. There were eight staff trained to facilitate the
primary service. The program had six therapeutic managers, a shift supervisor, and the clinical
director as trained facilitators of the primary service. Three therapeutic managers and the clinical
director conducted the monthly internal fidelity monitoring for the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. To enhance the continuity of the service due to unexpected facilitator absence
and provide more individualized instruction as needed during the service, when possible, Oak
Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance Abuse for
their primary service. The manual is broken down into twenty-five (25) sessions. Each session
explains the topic the therapist will cover and provides handouts for the youth to complete. The
primary service can be utilized for individual treatment or group treatment. The program holds
Seeking Safety groups on Tuesdays and Thursdays for one hour each session. The program
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
keeps their documentation for the primary service in a SPEP binder labeled "Seeking Safety". The
binder contains all attendance sheets, internal fidelity monitoring checklists, and group schedules.
The training certificates are held by the training coordinator at the program. All youth who are
admitted to the program receive the primary service which is held as an open group run year-
round.
According to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 4/26/2022 via Microsoft Teams. The group began
at 7:30a. The group went over the expectations. Then the facilitator went over the activity sheet
with the youth. The youth were each asked to answer each question on the worksheet as a group.
The youth were respectful and well-behaved. The facilitator then discussed how to write their
commitments down and how to create their commitments. The facilitator was knowledgeable of
the material and ensured the youth understood the content before moving on. Group ended at
8:30a.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022, shows the program did have facilitator turnover for Seeking Safety. Two therapeutic
managers left the program and one therapeutic manager abruptly lost her life due to COVID-19.
The facilitator turnover did affect the primary service; however, the program continued running
groups with another trained facilitator. The documentation showed there were six staff who
facilitated the primary service during the annual review period. The primary service is an open
group run year-round. There should be no more than fifteen youth in each session.
The attendance sheet dated for 12/29/2021 shows the same time and date, but different lesson
and youth. The clinical director stated the facilitator could have forgotten to fill in the date on the
sheet and tried to remember what day she ran the group. The attendance sheet dated for
10/3/2021 shows twenty youth in the group. There was a bracket around five of the youth with a
note stating the five youth were receiving a supplemental support session. The technical
assistance specialist informed the clinical director to have the facilitator put the five youth on
another attendance sheet and label that one as a supportive session. The technical assistance
specialist stated on face value, it looks like the facilitator was trying to have more than the fifteen
youth in the same session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement in an effort to reduce staff turnover. In addition, the program provides
positive incentives to improve employee motivation and dedication, leading to better employee
retention. Through this policy, employees receive credit for years of service, employment
referrals, and for contributing to the positive culture, teamwork environment and for going above
and beyond normal job expectations. Each employee who receives credit for doing these things is
entered in a monthly drawing at each facility. In the event of facilitator turnover, the co-facilitator
will assume the role of the facilitator for the group. If a co-facilitator is not present, a new qualified
facilitator for the primary service will take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022, shows there were thirty-two (32) internal fidelity monitorings completed for facilitators of
Seeking Safety. The program was missing a fidelity monitoring for one therapeutic manager for
the month of June 2021 and one fidelity monitoring for a therapeutic manager for the month of
December 2021. Four therapists and the clinical director conducted the monthly internal fidelity
monitoring during the annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the completed monthly internal fidelity checks for facilitators of Seeking Safety.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual evaluations for three therapeutic managers. The three evaluations
specifically address the staff member’s skill in delivering Seeking Safety.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 34 youth in the
sample, 97% (33 of 34) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 34 youth in the
sample, 100% (34 of 34) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 10 youth
High = 15 youth
Total Youth in Sample = 34 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 34 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (29 of 34) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 44% (15 of 34) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 91 95 96%
This SPEP report evaluates Seeking Safety, an intervention delivered at Oak Grove Academy.
36 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 34 total youth sampled, only
33 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 33
received at least the recommended weeks of service. Youth in the sample completed between 27
and 108 weeks of service, with an average of 41 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 34 total youth
sampled, all 34 included dosage in the EBS Module. Of those youth, all 34 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 43 and 112 hours of
service, with an average of 66 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and fidelity monitoring is completed every month for
each facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Oak Grove SS.pdf


state: FL

facility_name: Oak Grove Academy

summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Aggression Replacement Training
SPEP Review Period: April 1, 2022-March 31, 2023
SPEP Review Conducted: May 10, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2022 through March 31, 2023 shows
the program utilizes Aggression Replacement Training (ART) as their primary service. Aggression
Replacement Training is provided by a qualified trainer over a three-day period. There are five
staff members trained in the primary service- three therapeutic managers, the regional clinical
director, and the former clinical director. One of the therapeutic managers facilitated the primary
service during the review period. Each facilitator also takes turns conducting the monthly internal
fidelity monitoring. One therapeutic manager, the regional clinical director, and the former clinical
director conducted the monthly internal fidelity monitorings during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. For a
primary service without a formalized training, the facilitator will receive training from a licensed
clinician. This training will be documented in the facilitator’s On the Job Training Plan. The
facilitator will receive additional training on the curriculum as needed from the licensed clinician
documented within clinical supervision. All facilitators of a primary service will receive training
prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training (ART) manual. The manual is broken
down into six sections. The first section covers Skillstreaming. The second section deals with
anger control. The third section covers moral reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
use. It includes training step guides, fidelity forms and hassle logs, and the program materials.
The program provides the students with copies of handouts for each lesson. The program
conducts the primary service on Thursdays, Saturdays, and Sundays. The program keeps their
documentation for the primary service in a SPEP binder labeled "ART". The binder contains all
attendance sheets, internal fidelity monitoring checklists, training certificates, and group
schedules. A copy of the training certificates are held by the training coordinator at the program.
All youth who are admitted to the program receive ART. The service is held as a closed group.
According to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training was observed on 5/10/2023 via Microsoft Teams.
The group began at 7am. There were ten youth, two staff, and one facilitator present for the
group. The facilitator asked the youth to open the group. The lesson for the day is introducing the
ABCs of Anger Control. The facilitator discussed internal and external triggers. The facilitator
explained how the role plays were supposed to go. The facilitator was knowledgeable of the
material and presented the content in a way the youth could understand. The group ended at
8:02a.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows the program did have facilitator turnover; however, the turnover did not affect delivery of
Aggression Replacement Training (ART). The trained staff (therapeutic manager and former
clinical director) were not running a cohort at the time they left. The documentation showed there
was one therapeutic manager who facilitated ART during the annual review period.
There were four cohorts completed and one ongoing during the annual review period. The first
cohort began on 4/7/2022 with ten (10) youth and ended 6/12/2022 with all ten (10) youth. The
second cohort began on 6/23/2022 with ten (10) youth and ended 8/31/2022 with all ten (10)
youth. The third cohort began on 9/8/2022 with ten youth and ended 11/9/2022 with all ten (10)
youth. The fourth cohort began on 11/14/2022 with nine youth and ended 1/25/2023 with all nine
youth. The fifth cohort began on 2/12/2023 with ten (10) youth and had the same ten (10) youth at
the end of the review period.
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows there were thirty-six (36) internal fidelity monitorings completed for facilitators of
Aggression Replacement Training. A therapeutic manager, the regional director of clinical
services, and the former clinical director conducted the internal fidelity monitorings. The fidelity
monitoring on 7/17/2022 did not have several sections filled out. The form only had one or two
areas filled out and was signed by the facilitator on 7/17/2022. The area where including Likert
scale was not completed.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted nor taken on the completed monthly internal fidelity checks for Aggression
Replacement Training.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual evaluations with an employee performance evaluation addendum
for two therapeutic managers and two employee performance evaluation addendums for another
therapeutic manager and the current clinical director. All evaluations specifically address the staff
member’s skill in delivering Aggression Replacement Training. Two of the addenda were signed
and dated in April 2023 and one was signed and dated in May 2023.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 44 youth in the
sample, 80% (35 of 44) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 44 youth in the
sample, 100% (44 of 44) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 10 youth
Moderate-High = 16 youth
High = 14 youth
Total Youth in Sample = 44 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 32%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 14 youth
Total Youth in Sample = 44 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (40 of 44) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 32% (14 of 44) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 93 100 93%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Oak
Grove Academy.
63 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Duration. Of the 44 total youth sampled, all
44 included dosage with end dates in the EBS Module. Of those youth, 35 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 14 weeks of
service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 44 total youth
sampled, all 44 included dosage in the EBS Module. Of those youth, 44 received at least the
recommended hours of service. Youth in the sample completed 30 hours of service.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 98%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

file_name: FY2022-23 Oak Grove ART.pdf

program_name_raw: Oak Grove Academy

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Impact of Crime
SPEP Review Period: April 1, 2022-March 31, 2023
SPEP Review Conducted: May 10, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2022 through March 31, 2023 shows
the program utilizes the Impact of Crime Curriculum (IOC) as their primary service. The training is
a three-day, twenty-four (24)-hour training via Microsoft Teams which is delivered from a qualified
trainer. There were ten (10) staff trained to facilitate IOC- the former Director of Student Services,
three former Coach Counselors, the Program Director, two current Transition Service Managers,
the former Director of Group Living, the Regional Compliance Director, and a Case Manager. The
former Director of Student Services, the former Director of Group Living, a Case Manager, and
the Regional Compliance Director conducted the monthly internal fidelity monitorings for the
annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. For a
primary service without a formalized training, the facilitator will receive training from a licensed
clinician. This training will be documented in the facilitator’s On the Job Training Plan. The
facilitator will receive additional training on the curriculum as needed from the licensed clinician
documented within clinical supervision. All facilitators of a primary service will receive training
prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all seven
chapters of the service and provides a script for the facilitator to follow. The manual also has an
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
appendix and a student guide with all the student exercises. The program makes copies of the
student exercises and passes them out to the youth during each lesson. The program conducts
the primary service on Mondays and Wednesdays. The program keeps all SPEP documentation
(attendance sheets, fidelity checks, and copies of training certificates) in a binder labeled "IOC
SPEP documentation" in the case management office. All youth admitted to the program are
required to participate in the primary service. The primary service is a closed group with no more
than twelve youth in the cohort.
According to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed via Microsoft Teams on 5/3/2023. The group began at
4:32pm in the cafeteria. There were eleven (11) youth, two staff, and one facilitator present for the
group. The facilitator began the group with an icebreaker activity. The facilitator linked the
icebreaker with the purpose of the service. The facilitator was knowledgeable of the material and
was able to present it in a manner that the youth were able to understand. The group ended at
5:34p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows the program did have facilitator turnover for Impact of Crime (IOC) which negatively
affected service delivery, as the staff who left the program were running groups.
The program completed six cohorts during the review period and had one cohort ongoing at the
time of the review. The first cohort began on 11/18/2021 with nine youth and ended 4/13/2022
with six youth. Three youth were released from the program without completing the primary
service.
The second cohort began on 6/20/2022 with twelve (12) youth and ended 10/28/2022 with three
youth. The facilitator skipped chapter three objective three. The attendance sheet dated
7/20/2022 shows the facilitator was on chapter three objective two. The next attendance sheet,
dated 7/25/2022, shows the facilitator on chapter three objective four. There was a note stating
three youth received chapters six and seven in reverse order so they could ensure the youth
received the primary service prior to leaving the program. Three youth were released on
8/15/2022. One youth was released on 8/17/2022. One youth was released on 9/5/2022. One
youth was released on 10/10/2022. One youth was released on 10/17/2022. The facilitator who
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
began the cohort left the program. The Regional Compliance Director and a Case Manager took
over the cohort.
The third cohort began on 6/20/2022 with twelve (12) youth and ended 10/19/2022 with six youth.
There was one youth released on 8/10/2022. Another youth was released on 8/3/2022. A youth
was released on 8/29/2022. One additional youth was released on 10/5/2022. A case manager
took over the cohort when the facilitator left the program.
The fourth cohort began on 8/11/2022 with twelve (12) youth and ended 12/4/2022 with eleven
(11) youth. An attendance sheet dated 9/2/2022 shows the group ran from 6a to 8a. On
9/10/2022, the group ran from 6:57a to 8a. One youth was released on 10/14/2022. The Regional
Compliance Director began the cohort and a Case Manager facilitated seven sessions.
The fifth cohort began on 10/4/2022 with eight youth and ended 2/19/2023 with eight youth. The
sixth cohort began on 11/23/2022 with eleven (11) youth and ended 3/22/2023 with all eleven (11)
youth. The seventh cohort began on 1/10/2023 with eleven (11) youth and had the same eleven
(11) youth at the time of the annual review.
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows there were twenty-eight (28) internal fidelity monitorings completed for facilitators of Impact
of Crime. The program was missing an internal fidelity monitoring checklist for the month of
January 2023 for a Transition Service Manager. The former Director of Student Services, the
former Director of Group Living, a Case Manager, and the Regional Compliance Director
conducted the monthly internal fidelity monitorings for the annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted nor taken on the completed monthly internal fidelity checks for facilitators of Impact
of Crime.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an annual performance evaluation for two Transition Service Managers.
The evaluations had an addendum added which specifically addressed the staff member’s skill in
delivering Impact of Crime.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 61 youth in the
sample, 61% (37 of 61) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 61 youth in the
sample, 70% (43 of 61) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 13 youth
Moderate-High = 21 youth
High = 21 youth
Total Youth in Sample = 61 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 34%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 21 youth
Total Youth in Sample = 61 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (55 of 61) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 34% (21 of 61) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 72 85 85%
This SPEP report evaluates Impact of Crime, an intervention delivered at Oak Grove Academy.
63 youth completed the program during the review period. Of those youth, 61 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 61 total youth sampled, only
47 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 37
received at least the recommended weeks of service. Youth in the sample completed between 7
and 21 weeks of service, with an average of 16 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 61 total youth
sampled, only 52 included dosage in the EBS Module. Of those youth with correct dosage, 43
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
received at least the recommended hours of service. Youth in the sample completed between 11
and 44 hours of service, with an average of 32 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and fidelity monitoring is completed every month for
each facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 Oak Grove IOC.pdf

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Seeking Safety
SPEP Review Period: April 1, 2022-March 31, 2023
SPEP Review Conducted: May 10, 2023
Florida Department of Juvenile Justice
Report Date: 9/14/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2022 through March 31, 2023 shows
the program utilizes Seeking Safety as their primary service. Seeking Safety training is conducted
by the clinical director as an in-service training. There were six staff trained to facilitate the
primary service- four therapeutic managers, the former clinical director, and the clinical director.
The current clinical director, two therapeutic managers, the former clinical director, and the
regional director of clinical services conducted the monthly internal fidelity monitorings for the
annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. For a
primary service without a formalized training, the facilitator will receive training from a licensed
clinician. This training will be documented in the facilitator’s On the Job Training Plan. The
facilitator will receive additional training on the curriculum as needed from the licensed clinician
documented within clinical supervision. All facilitators of a primary service will receive training
prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance Abuse for
their primary service. The manual is broken down into twenty-five sessions. Each session
explains the topic the therapist will cover and provides handouts for the youth to complete. The
service can be utilized for individual treatment or group treatment. The program keeps all relevant
documentation in a SPEP binder labeled "Seeking Safety". The binder contains all attendance
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
sheets, internal fidelity monitoring checklists, and group schedules. The training certificates are
held by the training coordinator at the program.
The program holds Seeking Safety as an open group on Tuesdays and Thursdays for one hour
each session. All youth who are admitted to the program receive the primary service.
According to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 4/26/2023. The group was held in Cottage A and
began at 1:34p. There were fourteen (14) youth, two staff, one facilitator, and the Clinical Director
present for the group. The group began with check in. The facilitator asked the youth what they
were going to improve upon. The facilitator discussed healthy relationships. There were a few
youth who were laying back on the chairs with their eyes closed. One of the staff in the room
noticed quickly and woke the youth up. The youth began to get loud toward the end of the group.
The facilitator was knowledgeable of the material and ensured the youth understood the content
before moving on. The group ended at 2:35p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows the program experienced facilitator turnover which adversely affected delivery of Seeking
Safety; however, the program continued the primary service with another trained facilitator.
A therapeutic manager left the program in June 2022 and the former clinical director left the
program in September 2022. Seeking Safety is an open group run year-round with no more than
fifteen (15) youth in each session.
The program conducted the primary service each month. There were thirty-eight (38) dates
between April 2022 and July 2022 where the program had more than fifteen (18) youth in the
Substance Abuse Overlay Service (SAOS) group. The facilitators documented fifteen (15) youth
received the traditional group for one hour and the additional youth would receive fifteen (15)
minutes of supportive counseling. The technical assistance specialist informed the clinical director
there should only be fifteen (15) youth in a group. The additional youth should receive the full hour
in a separate group instead of only fifteen (15) minutes of supportive counseling.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
There were twelve (12) attendance sheets between August 2022 and March 2023 which showed
the program conducted twenty (20)-minute supportive sessions with up to three youth. The
technical assistance specialist informed the new clinical director all youth need to get the full hour
of group. The youth making up a group session should receive a full hour as well, instead of a
twenty-minute supportive session.
According to the program's Standardized Program Evaluation Protocol procedure, in the event of
facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2022 through March 31, 2023
shows there were seventeen (17) internal fidelity monitorings completed for facilitators of Seeking
Safety. The current clinical director, two therapeutic managers, the former clinical director, and the
regional director of clinical services conducted the monthly internal fidelity monitorings for the
annual review period. The program completed all internal fidelity monitorings for each month for
each facilitator.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the completed monthly internal fidelity checks for facilitators of Seeking Safety.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
is completing a written warning and placing the facilitator on a performance improvement plan.
The plan will be reviewed with the facilitator by the qualified facilitator conducting the monitoring
and facilitator’s supervisor. In the event an issue identified during the primary service is
detrimental to the youth’s safety and /or treatment, the matter will be taken up directly with the
program’s Human Resource Manager and be addressed in accordance with the employee
handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual evaluations with an employee performance evaluation addendum
for two therapeutic managers and two employee performance evaluation addendums for another
therapeutic manager and the current clinical director. All evaluations specifically address the staff
member’s skill in delivering Seeking Safety. Two of the addenda were signed and dated in April
2023 and one was signed and dated in May 2023.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 59 youth in the
sample, 93% (55 of 59) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 59 youth in the
sample, 98% (58 of 59) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 13 youth
Moderate-High = 20 youth
High = 21 youth
Total Youth in Sample = 59 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 36%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 21 youth
Total Youth in Sample = 59 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (54 of 59) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 36% (21 of 59) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 89 95 94%
This SPEP report evaluates Seeking Safety, an intervention delivered at Oak Grove Academy.
63 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 59 total youth sampled, all
59 included dosage with end dates in the EBS Module. Of those youth, 55 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 59 weeks of
service, with an average of 35 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 59 total youth
sampled, all 59 included dosage in the EBS Module. Of those youth, 58 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 19 and 78 hours of
service, with an average of 56 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2022-23 Oak Grove SS.pdf

service_type: Seeking Safety