summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Impact of Crime
SPEP Review Period: April 1, 2020-March 31, 2021
SPEP Review Conducted: March 29, 2021
Florida Department of Juvenile Justice
Report Date: 6/15/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 2020 through March 2021 shows
there are seven staff trained to facilitate the primary service. Due to COVID-19, the training is a
three-day training via Microsoft Teams which is delivered from a qualified trainer. One trained staff
member is a case manager, the other six are all floor staff. The floor staff supervisor conducted
the fidelity monitoring during the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services. Training for individual counseling services during On-the-
Job Training will be provided by a licensed clinician consisting of a review of common issues such
as trauma, mental health, substance abuse, violence victimization, impulsivity, and relationships
prevalent in the population being served. In addition, during On-the-Job Training, the procedures
for conducting the individual sessions including location, scheduling, and safety and security
concerns are covered.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the curriculum Impact of Crime: Addressing the Harm v 2.0. The manual
provides step by step procedures for implementing the program. There are scripts and activities
for the facilitators to use. There are no student manuals given to the youth. The facilitators make
copies of the worksheets and pass them out to the youth during group. The primary service is
held on Tuesdays and Thursdays for one hour each session. The program keeps their
documentation for the primary service in a SPEP binder labeled Impact of Crime. The binder
contains all attendance sheets, internal fidelity monitoring checklists, and group schedules. The
training certificates are held by the training coordinator at the program. All youth who are admitted
to the program receive the primary service.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 3/24/2021 via FaceTime. The group began at
8:34a. The facilitator began with a review of the group rules. The facilitator then broke the group
up into two different groups to complete the icebreaker. The facilitator asked the youth to recap
what they learned from the beginning of chapter two. The facilitator read the exercise from the
manual and asked the youth to answer the questions. The group then discussed making amends,
apologies, and community service. The facilitator assigned homework for the youth to complete
prior to the next session. The group ended at 9:31a. The facilitator followed the lesson plan.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program did not have facilitator turnover for Impact of
Crime (IOC). There were six cohorts completed during the annual review period, with three
additional cohorts running at the time of the annual review. The cohorts began in December 2020,
November 2020, and February 2021. The attendance sheets were not clear with the completion
date for two of the cohorts. The dates of the cohorts are: 12/20/2020-current, 11/29/2020-current,
2/25/2020-unknown completion date, 2/28/2021-current, 2/25/2020-8/13/2020, and 7/27/2020-
unknown completion date.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The cohort which began on 2/25/2020 and ended on 8/13/2020 began with eight youth and ended
with four. Thirty-one (31) sessions were conducted on back-to-back days. There was a gap in
service for over one month from 5/21/2020 to 6/29/2020.
The cohort which began on 2/25/2020 had an unknown end date. A review of the Evidence-Based
Services (EBS) module suggests this cohort ended 8/13/2020 with four youth. There were
attendance sheets dated 12/22/2019 with the same youth listed on the 2/25/2020 attendance
sheets. The attendance sheets on went to 1/1/2020 and then there was a gap until 2/25/2020.
There was one facilitator listed on the attendance sheets from 12/22/2019 through 1/1/2020 and
then a new facilitator took over on 2/25/2020. The attendance sheets showed the cohort restarted
the curriculum. There were only nine attendance sheets for the cohort beginning on 2/25/2020.
The dates of the attendance sheets are 2/25/2020, 3/6/2020, 3/31/2020, 4/2/2020, 4/7/2020,
5/14/2020, 5/16/2020, 7/11/2020, and 7/16/2020. The clinical director stated the original facilitator
went out on medical leave. This prompted the change in facilitators for this cohort.
The cohort which began on 7/27/2020 began with twelve youth. The attendance sheets did not
document the date of the last session and the completion date is unclear in the EBS module.
Three youth are missing end dates; two have an end date of 11/23/20 with a note they were
released prior to completing the group. The last attendance sheet provided by the program had
eight youth. The cohort was ran back to back on nine occassions. There was a gap in service of
over one month from 9/8/2020 through 10/21/2020.
The cohort which began on 11/29/2020 is currently ongoing. The cohort began with five youth.
The cohort was ran back to back on eight occassions. There was a gap in service for one month.
The documentation showed the first attendance sheet was dated on 11/29/2020. The next
attendance sheet was dated on 12/30/2020. The documentation also showed another gap
between 1/30/2021 and 2/24/2021.
The cohort which began on 12/20/2020 is currently ongoing. The group began with six youth.
There were two youth added to the cohort shortly after beginning, as temporarily allowed by the
Department during the COVID-19 pandemic. The documentation then showed the cohort went
back to six youth with no explanation.
The cohort which began on 2/28/2021 and is currently running began with ten youth. The cohort
was ran back to back on 3/1/2020 and 3/2/2020 and then again on 3/13/2020 and 3/14/2020.
The technical assistance specialist spoke with the case manager in charge of the primary
service’s documentation. The program has the youth separated by cottage. Both cottages are in
the primary service and are on the same lesson. The service is held in the cafeteria where both
cottages can be separated, but continue to receive the lessons together.
Of the six cohorts run during the annual review period, two cohorts' attendance sheets listed three
facilitators. The technical assistance specialist spoke with the case manager who is in charge of
the IOC documents and stated there should only be one or two staff members listed unless there
were three staff facilitating the specific cohort. The case manager stated the facilitators were listed
just in case one facilitator was out and unable to facilitate the groups. The trained staff rotated
days they facilitated the primary service. Technical assistance specialist informed the case
manager the facilitator assigned to the cohort should be consistent. This consistency is best for
the youth as it allows stability during the primary service. Changing the facilitators out each
session does not allow the youth stability which decreases the effectiveness of the primary
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
incentives, which in their experience has shown to improve employee motivation and dedication
and lead to better employee retention. Through this policy, employees receive credit for years of
service, employment referrals, and for contributing to the positive culture, teamwork environment
and for going above and beyond normal job expectations. Each employee who receives credit for
doing these things is entered in a monthly drawing at each facility. In the event of facilitator
turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-facilitator is
not present, a new qualified facilitator for the primary service will take over conducting the service.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 2020 through March 2021
shows there were four internal fidelity monitorings completed. The dates of the internal fidelity
checks were completed on 3/1/2021, 2/25/2021, 1/24/2021, and 11/22/2020. The case manager
in charge of the primary service’s documentation stated the program did not have enough trained
facilitators to conduct internal fidelity checks prior to November 2020.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions. For the primary service of individual counseling, fidelity monitoring is conducted by
licensed clinician in the review of therapy notes to verify provision of effective and appropriate
intervention in line with the treatment plan and review of trackers to ensure compliance with
frequency and duration.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the four completed monthly internal fidelity checks.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified are addressed continue to be an issue, the next stage is
completing a written warning and placing the facilitator on a performance improvement plan. The
performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program did not provide any documentation of annual staff evaluations for the annual review
period of April 2020 through March 2021. The clinical director stated the program has an
addendum for the annual evaluations which addresses facilitator skill delivering the primary
service, but none were provided for review.
According to the program’s Standardized Program Evaluation Protocol, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from fidelity monitoring tools as well as an assessment of the
facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 34 youth in the
sample, 24% (8 of 34) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 34 youth in the
sample, 15% (5 of 34) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 8 youth
Moderate-High = 11 youth
High = 10 youth
Total Youth in Sample = 34 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 29%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 10 youth
Total Youth in Sample = 34 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (29 of 34) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 29% (10 of 34) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 18 25 40%
Totals 50 85 59%
This SPEP report evaluates Impact of Crime, an intervention delivered at Oak Grove Academy.
36 youth completed the program during the review period. Of those youth, 34 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 34 total youth sampled, only
20 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 8
received at least the recommended weeks of service. Youth in the sample completed between 3
and 25 weeks of service, with an average of 14 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 34 total youth
sampled, only 26 included dosage in the EBS Module. Of those youth with correct dosage, 5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
received at least the recommended hours of service. Youth in the sample completed between 6
and 4 hours of service, with an average of 18 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 67%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring there
are no unnecessary gaps in service, fidelity monitoring is completed every month for each
facilitator, and each facilitator’s performance evaluation includes discussion of their skill in
delivering this primary service.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11


state: FL

facility_name: Oak Grove Academy

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, INC.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Seeking Safety
SPEP Review Period: April 1, 2020-March 31, 2021
SPEP Review Conducted: April 2, 2021
Florida Department of Juvenile Justice
Report Date: 6/15/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 1, 2020 through March 31, 2021
shows there are seven staff trained in Seeking Safety. The training is conducted by the clinical
director as an in-service training. The program has five therapeutic mangers, a shift supervisor,
and the clinical director as trained facilitators of the primary service. All trained staff conduct the
monthly internal fidelity checks.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training from a qualified trainer. In order
to enhance the continuity of the service due to unexpected facilitator absence and provide more
individualized instruction as needed during the service, when possible, Oak Grove Academy will
make all efforts to have a co-facilitator for the primary service. The co-facilitator will have
completed a formalized training for the primary service from a qualified trainer. For a primary
service without a formalized training, the facilitator will receive training from a licensed clinician.
This training will be documented in the facilitator’s On the Job Training Plan. The facilitator will
receive additional training on the curriculum as needed from the licensed clinician documented
within clinical supervision. All facilitators of a primary service will receive training prior to
conducting the services. Training for individual counseling services during On-the-Job Training
will be provided by a licensed clinician consisting of a review of common issues such as trauma,
mental health, substance abuse, violence victimization, impulsivity, and relationships prevalent in
the population being served. In addition, during On-the-Job Training, the procedures for
conducting the individual sessions including location, scheduling, and safety and security
concerns are covered.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The program utilizes the Seeking Safety: A Treatment Manual for PTSD and Substance Abuse for
their primary service. The manual is broken down into twenty-five sessions. Each session
explains the topic the therapist will cover and provides handouts for the youth to complete. The
primary service can be utilized for individual treatment or group treatment. The program holds
Seeking Safety groups on Tuesdays and Thursdays for one hour each session. The program
keeps their documentation for the primary service in a SPEP binder labeled Seeking Safety. The
binder contains all attendance sheets, internal fidelity monitoring checklists, and group schedules.
The training certificates are held by the training coordinator at the program. All youth who are
admitted to the program receive Seeking Safety. The primary service is an open group run year-
round.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was observed on 3/24/2021 via FaceTime. The group began at
3:32p. The facilitator informed the youth about the technical assistance specialist observing
group. The lesson for the group wass Getting Others to Support Your Recovery. The facilitator
asked a youth to lead the check in. The facilitator then went over the group rules with the youth.
The facilitator read statements and the lesson from the facilitator manual. The youth were asked
to write down the names of people who are supportive of their recovery. Each youth took turns
reading a letter from the manual and then discussing it. The group concluded at 4:16p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 2020 through March 2021
shows the program had facilitator turnover which adversely affected delivery of Seeking Safety.
The documentation showed there were six staff who facilitated Seeking Safety during the annual
review period. The documentation shows there are currently three trained staff facilitating the
primary service as of January 2021. The other three trained facilitators are still employed at the
program, but they are not actively facilitating the primary service. The primary service is an open
group run year-round.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
incentives which, in their experience, has shown to improve employee motivation and dedication
and lead to better employee retention. Through this policy, employees receive credit for years of
service, employment referrals, and for contributing to the positive culture, teamwork environment
and for going above and beyond normal job expectations. Each employee who receives credit for
doing these things is entered in a monthly drawing at each facility. In the event of facilitator
turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-facilitator is
not present, a new qualified facilitator for the primary service will take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed shows there were twenty-eight internal fidelity monitorings
completed, accounting for eleven (11) of the twelve-month review period. The program did not
have any internal fidelity checks completed during the month of June 2020. The program was
missing internal fidelity checks for the following months: March 2021 for two therapists, December
2020 for one therapist, November 2020 for one therapist, August 2020 for one therapist, July
2020 for three therapists, and May 2020 for one therapist. All trained staff will perform monthly
internal fidelity checks.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the monthly internal fidelity checks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Annual evaluations for facilitators of Seeking Safety were not provided. Addenda were provided
for review for four Therapeutic Managers; each was a copy/paste and did not specifically address
their skill in delivering Seeking Safety. Additionally, all addenda were signed after the end of the
review period and the reviewer's initial request for documentation.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 30 youth in the
sample, 33% (10 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 40 contact hours for this type of service. Of the 30 youth in the
sample, 43% (13 of 30) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 7 youth
Moderate-High = 10 youth
High = 9 youth
Total Youth in Sample = 30 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 9 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (26 of 30) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 30% (9 of 30) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 66 95 69%
This SPEP report evaluates Seeking Safety, an intervention delivered at Oak Grove Academy.
36 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 30 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 10
received at least the recommended weeks of service. Youth in the sample completed between 28
and 65 weeks of service, with an average of 41 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 30 total youth
sampled, all 30 included dosage in the EBS Module. Of those youth, 13 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 9 and 131 hours of
service, with an average of 39 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 75%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, fidelity monitoring is completed every month for each
facilitator, and each facilitator’s performance evaluation includes discussion of their skill in
delivering this primary service.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Oak Grove Academy

program_status_note: Open

service_type: Seeking Safety


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Aggression Replacement Training
SPEP Review Period: April 1, 2021-March 31, 2022
SPEP Review Conducted: May 4, 2022
Florida Department of Juvenile Justice
Report Date: 7/27/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2021, through March 31, 2022, shows
the program utilizes Aggression Replacement Training (ART) as their primary service. ART
Training is provided by a qualified trainer over a three-day period. There are eight staff members
trained in the primary service: six therapeutic managers, one shift supervisor, and the clinical
director. Three of the eight trained staff members facilitated the primary service during the review
period. Each facilitator also takes turns conducting the monthly internal fidelity monitoring. Three
therapeutic managers and the clinical director conducted monthly internal fidelity monitoring.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. To enhance the continuity of the service due to unexpected facilitator absence
and provide more individualized instruction as needed during the service, when possible, Oak
Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training (ART) manual. The manual is broken
down into six sections. The first section covers Skillstreaming. The second section deals with
anger control. The third section covers moral reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
use. It includes training step guides, fidelity forms and hassle logs, and the program materials.
The program provides the students with copies of handouts for each lesson.
The program keeps their documentation for the primary service in a SPEP binder labeled ART.
The binder contains all attendance sheets, internal fidelity monitoring checklists, training
certificates, and group schedules. A copy of the training certificates is held by the training
coordinator at the program. All youth who are admitted to the program receive ART. Groups are
held on Thursdays, Saturdays, and Sundays, and is a closed group.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training was observed on 4/21/2022 via Microsoft Teams.
The group began at 6:23p. The facilitator went over group rules. The lesson for the day is getting
ready for a difficult conversation. The facilitator had a youth lead the group. The youth read the
script from the facilitator’s manual. The facilitator reminded the youth to look for each skill step in
the youth’s role plays. The facilitator did not assign a skill step for the youth to observe. The
assigned homework was to practice the skill. The youth completed the first three questions on the
homework sheet. The group ended at 7:40p.
The technical assistance specialist spoke with the facilitator, the program director, the clinical
director, and the regional clinical director to remind staff the youth cannot lead groups. The
facilitator must lead the group since they are the ones trained in the service. Having a youth lead
the group is the same as having an untrained staff run the group. The facilitator stated she would
not have a youth lead the group again. The facilitator stated she usually has a youth assist with
the group to help them with their leadership. The clinical director stated the facilitator has not had
a youth run the group when she was conducting the fidelity monitoring. The clinical director also
stated the facilitator could have been nervous because she was being observed.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022, shows the program had turnover of facilitators for Aggression Replacement (ART). The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
facilitator turnover did not affect delivery of ART as the three trained staff (therapeutic managers)
were not running a cohort at the time they left. The documentation showed there were three
therapeutic managers who facilitated the primary service during the annual review period. There
were three cohorts completed during the annual review period. The dates of the cohorts are
2/18/2021-5/2/2021, 9/9/2021-11/21/2021, and 12/9/2021-3/13/2022. ART is a closed group run
year-round.
The 2/18/2021 cohort began and ended with nine youth. There were several attendance sheets
which did not document the lesson being taught during the group. There were also several
attendance sheets which were not documented on the appropriate form.
There were no attendance sheets provided for 5/2/2021-9/8/2021. The clinical director stated the
program was waiting until they receive more admissions before beginning a new cohort. There
were not enough youth at the program with aggression as one of their top three criminogenic
needs during the time between 5/2/2021 and 9/8/2021.
The 9/9/2021 cohort began with ten (10) youth and ended with seven youth. The 12/9/2021 cohort
began with ten (10) youth and ended with nine youth. There was a note from the clinical director
stating there was a gap in service from 12/20/2021-1/19/2022 due to a staff member testing
positive for COVID. The youth were separated as a precaution.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety (90) days. The process encourages personal growth and development and
recognizes areas of improvement in an effort to reduce staff turnover. In addition, the program
provides positive incentives, to improve employee motivation and dedication. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022, shows there were twenty-one (21) internal fidelity monitorings completed for facilitators of
Aggression Replacement Training. The program missed the Skillstreaming fidelity for the month of
January. Four therapeutic managers and the clinical director conducted the internal fidelity
monitorings.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service
monthly. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or an internal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions. For the primary service of individual counseling, fidelity monitoring is conducted by
licensed clinician in the review of therapy notes to verify provision of effective and appropriate
intervention in line with the treatment plan and review of trackers to ensure compliance with
frequency and duration.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the completed monthly internal fidelity checks of Aggression Replacement
Training.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual evaluations for three therapeutic managers. The three evaluations
specifically address the staff member’s skill in delivering Aggression Replacement Training.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 16 youth in the
sample, 88% (14 of 16) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 16 youth in the
sample, 94% (15 of 16) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 81%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 1 youth
Moderate-High = 3 youth
High = 9 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 56%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 81% (13 of 16) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 56% (9 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 91 100 91%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Oak
Grove Academy.
36 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Duration. Of the 16 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended weeks of service. Youth in the sample completed between 9
and 35 weeks of service, with an average of 12 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, all 16 included dosage in the EBS Module. Of those youth, 15 received at least the
recommended hours of service. Youth in the sample completed b30 hours of service.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring the
facilitator adheres to the manual and fidelity monitoring is completed every month for each
facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Oak Grove ART.pdf

service_type: Aggression Replacement Training


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Impact of Crime
SPEP Review Period: April 1, 2021-March 31, 2022
SPEP Review Conducted: May 4, 2022
Florida Department of Juvenile Justice
Report Date: 7/27/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2021, through March 31, 2022, shows
the program utilizes the Impact of Crime Curriculum (IOC) as their primary service. The training is
a three-day, twenty-four-hour training via Microsoft Teams, delivered from a qualified trainer.
There were six staff trained to facilitate the primary service. The trained staff members are one
case manager/transition coordinator, four coach counselors, and one shift supervisor. A coach
counselor, a shift supervisor, and the case manager/transition coordinator conducted the monthly
internal fidelity monitoring for the annual review period.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. To enhance the continuity of the service due to unexpected facilitator absence
and provide more individualized instruction as needed during the service, when possible, Oak
Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime Curriculum. Then the manual goes over all seven chapters
of the service and provides a script for the facilitator to follow. The manual also has an appendix
and a student guide with all the student exercises. The program makes copies of the student
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
exercises and passes them out to the youth during each lesson. The program conducts the
primary service on Mondays and Wednesdays. The program keeps all SPEP documentation
(attendance sheets, fidelity checks, and copies of training certificates) in a binder labeled "IOC
SPEP documentation" in the case management office. All youth admitted to the program are
required to participate in IOC. The primary service is a closed group with no more than twelve (12)
youth in each cohort.
According to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance with its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed. The program finished a cohort the
middle week of April 2022. The technical assistance specialist scheduled an observation from the
last group session for 4/11/2022, but the facilitator called out sick that day. The program finished
the cohort later that week. This indicator is rated as non-applicable.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022 shows the program did have turnover of Impact of Crime facilitators which affected service
delivery. There were several cohorts where the program had two trained staff listed on the
attendance sheets who alternated facilitating the service during those cohorts. There were four
coach counselors trained in the primary service who left the program. There were five cohorts
completed and one unknown cohort during the annual review period. The dates of the cohorts are
12/20/2020-5/11/2021, 2/28/2021-9/5/2021, 7/15/2021-11/16/2021, 7/15/2021-11/10/2021,
11/28/2021-middle of April 2022, and an unknown cohort. The primary service is a closed group
run year-round.
The 12/20/2020 cohort began with six youth and ended with five youth. The documentation shows
the group was ran back-to-back on 4/3/2021-4/4/2021 and 4/10/2021-4/11/2021.
The 2/28/2021 cohort began with ten (10) youth and ended with four youth. The program did not
have documentation of any attendance sheets after 5/18/2021 for this cohort. There was a gap in
service for twenty-one days between 7/7/2021 and 7/28/2021. The attendance sheet dated for
7/30/2021 shows the cohort completed chapter seven objective four. The next attendance sheet
was dated for 9/5/2021. There was a gap of thirty-seven days. The clinical director and program
director stated the facilitator was trying to complete the community service project. The program
was having issues getting the community service project started/completed due to COVID. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
facilitator tried to line up a project outside of the program, but that fell through. The facilitator
decided to have the youth work on a project at the program on 9/5/2021.
The attendance sheets for both cohorts which began on 7/15/2021 showed the service was being
conducted to the fidelity of the model. One of the cohorts which began on 7/15/2021 started with
eight youth and ended with six youth. One youth was released on 9/13/2021 and another youth
was released on 11/10/2021. The other cohort which began on 7/15/2021 began with nine youth
and ended with six youth. One youth was transferred on 9/15/2021 and two other youth were
released on 9/15/2021 and 11/5/2021.
The cohort which began on 11/18/2021 began with nine youth and ended with nine youth. The
program ran the service on back-to-back days on 3/14/2022 and 3/15/2022.
The program stated there was another cohort run by a coach counselor who is no longer
employed with the program. The clinical director stated she lost all of the attendance sheets for
the cohort. The clinical director stated the coach counselor handed her the attendance sheets
before he left, but she misplaced them. There were internal fidelity checks for the coach counselor
for November 2021 and December 2021, but no attendance sheets were provided.
The attendance sheets for the cohorts lists a facilitator, co-facilitator, and sometimes a third staff
member; however, the program only had one staff member conduct the primary service for each
lesson. The case manager stated the facilitators were listed just in case one facilitator was out
and unable to facilitate the groups. The trained staff rotated days they facilitated the primary
service. The technical assistance specialist informed the case manager and the clinical director
the facilitator assigned to the cohort should be consistent. This consistency is best for the youth
as it allows stability during the primary service. Changing the facilitators out each session does
not allow the youth stability which decreases the effectiveness of the primary service.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development and recognizes
areas of improvement in an effort to reduce staff turnover. In addition, the program provides
positive incentives to improve employee motivation and dedication. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 1, 2021, through March 31,
2022, shows there were thirteen (13) internal fidelity monitorings completed for facilitators of
Impact of Crime. The program did not have internal fidelity monitoring for the shift supervisor for
the month of September 2021. The shift supervisor conducted the final session of the cohort on
9/5/2021. A coach counselor, shift supervisor, and the case manager/transition coordinator
conducted the monthly internal fidelity monitoring during the annual review period.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions. For the primary service of individual counseling, fidelity monitoring is conducted by
licensed clinician in the review of therapy notes to verify provision of effective and appropriate
intervention in line with the treatment plan and review of trackers to ensure compliance with
frequency and duration.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the monthly internal fidelity checks for facilitators of Impact of Crime.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an annual evaluation for the case manager/transition coordinator. The
evaluation specifically addressed the staff member’s skill in delivering Impact of Crime.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 35 youth in the
sample, 63% (22 of 35) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 35 youth in the
sample, 86% (30 of 35) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 15 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 43%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 15 youth
Total Youth in Sample = 35 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (30 of 35) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 43% (15 of 35) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Oak Grove Academy.
36 youth completed the program during the review period. Of those youth, 35 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 35 total youth sampled, only
32 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 22
received at least the recommended weeks of service. Youth in the sample completed between 10
and 37 weeks of service, with an average of 17 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 35 total youth
sampled, only 33 included dosage in the EBS Module. Of those youth with correct dosage, 30
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
received at least the recommended hours of service. Youth in the sample completed between 16
and 44 hours of service, with an average of 32 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and fidelity monitoring is completed every month for
each facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring dosage for all
youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2021-22 Oak Grove IOC.pdf

service_type: Impact of Crime