state: FL

facility_name: Oak Grove Academy

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Aggression Replacement Training
SPEP Review Date(s): 6/1/2020
Florida Department of Juvenile Justice
Report Date(s): 6/2/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 2019 through March 2020 shows the
program has four staff who were running Aggression Replacement Training groups. The
documentation provided shows there are three staff members who completed their training prior
to facilitating groups. One of the staff members was trained on 6/26/19 but ran a group on
6/21/19.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training manual. The manual is broken down
into six sections. The first section covers Skillstreaming. The second section deals with anger
control. The third section covers moral reasoning. Each of the first three sections has a script for
the facilitator to use for each lesson. Sections four through six are for the facilitator's use. It
includes training step guides, fidelity forms and hassle logs, and the program materials.
Pursuant to the facility's Standardized Program Evaluation Protocol, each primary service will be
facilitated in accordance to its manual. The primary service will be conducted as indicated in the
manual/protocol; this includes adherence to the session plan, group size, population, frequency,
needed materials, duration, and interventions. The facilitator of the primary service will specify the
topic covered, interventions utilized, duration of the service, youth present, and progress for each
session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training was unable to be observed due to the COVID-19
pandemic; therefore, this indicator is rated as non-applicable.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did have staff turnover; however, the turnover did not adversely affect the delivery of
Aggression Replacement Training (ART). There was a gap in services from mid-August until mid-
October, 2019. The clinical director stated three of the staff members were conducting groups
incorrectly. The three staff members were not running the groups in sequential order. The clinical
director said the groups were stopped in August 2019 to have all facilitators retrained in the
curriculum. The clinical director stated the program restarted ART groups at the end of October
2019 with one therapeutic manager conducting those groups. The program opened in March
2019. The program did not have enough youth available to run groups until May 2019.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
incentives, which in our experience has shown to improve employee motivation and dedication
and thus leads to better employee retention. Through this policy, employees receive credit for
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee that
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 2019 through March 2020 show
there were nine internal fidelity monitorings completed. Those fidelity monitorings were completed
on 7/24/2019, 1/19/2020, 1/25/2020, 1/31/2020, 2/27/2020, 2/29/2020, 3/19/2020, 3/21/2020, and
3/28/2020. There were no other fidelity monitorings completed.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions. For the primary service of individual counseling, fidelity monitoring is conducted by
licensed clinician in the review of therapy notes to verify provision of effective and appropriate
intervention in line with the treatment plan and review of trackers to ensure compliance with
frequency and duration.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the nine completed fidelity checks. There was missing fidelity monitoring
documentation in the months preceeding the gap in service, wherein the monitor noticed
discrepancies and sent facilitators out to be retrained.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If, during additional
fidelity monitoring, the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event an issue identified during the primary service is detrimental to the youth’s safety and
/or treatment, the matter will be taken up directly with the program’s Human Resource Manager
and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of March 2019 through February 2020
shows facilitator evaluations were completed. The facilitator evaluations had an area which
addressed facilitator skill delivering evidence-based practices. All facilitators trained in Aggression
Replacement Training had evaluations which specifically addressed their performance.
Pursuant to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 10 weeks for this type of service. Of the 10 youth in the
sample, 0% (0 of 10) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 30 contact hours for this type of service. Of the 10 youth in the
sample, 0% (0 of 10) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 5 youth
Total Youth in Sample = 10 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 10 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (8 of 10) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 50% (5 of 10) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 20 25 80%
Totals 65 100 65%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Oak
Grove Academy.
18 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Duration. Of the 10 total youth sampled, all
10 included dosage with end dates in the EBS Module. Of those youth, 0 received at least the
recommended weeks of service. Youth in the sample completed between 6 and 8 weeks of
service, with an average of 7 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 10 total youth
sampled, all 10 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed between 6 and 8 hours of service,
with an average of 7 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 70%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring all
facilitators receive formal training prior to delivering services to youth, by ensuring there are no
unnecessary gaps in service, and by ensuring fidelity monitoring is completed every month for
each facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Oak Grove Academy

program_status_note: Open

service_type: Aggression Replacement Training


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Impact of Crime
SPEP Review Date(s): 6/1/2020
Florida Department of Juvenile Justice
Report Date(s): 6/1/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 2019 through March 2020 showed
there were three staff trained to facilitate Impact of Crime. Each of the three staff members
received the four-day in person training from a qualified trainer prior to delivering services to the
youth.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services. Training for individual counseling services during On the
Job Training will be provided by a licensed clinician consisting of a review of common issues such
as trauma, mental health, substance abuse, violence victimization, impulsivity, and relationships
prevalent in the population being served. In addition, during On the Job Training, the procedures
for conducting the individual sessions including location, scheduling, and safety and security
concerns are covered.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm v 2.0. The manual provides step
by step procedures for implementing the program. There are scripts and activities for the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
facilitators to use. There are no student manuals given to the youth. The facilitators make copies
of the worksheets and pass them out to the youth during group.
The program used the incorrect sign in sheets during their first cohort beginning in May 2019. The
program identified the issue and began using the correct sign in sheets in June 2019.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed due to the COVID-19 pandemic;
therefore, this indicator is rated as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did have facilitator turnover; however, the turnover did not adversely affect delivery
of Impact of Crime. There was one staff member who left the program in September 2019, during
a cohort.The co-facilitator took over the group without any gaps in service. The program began
running Impact of Crime in May 2019 when the program received enough youth to begin a cohort.
The documentation does not reveal any gaps in service.
Three cohorts were completed during the annual review period of April 2019 through March 2020.
Two cohorts began in February 2020 and are currently ongoing.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
incentives, which in our experience has shown to improve employee motivation and dedication
and thus leads to better employee retention. Through this policy, employees receive credit for
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 2019 through March 2020 show
there were no internal fidelity monitorings completed. The clinical director stated two cohorts are
being run at the same time and, unfortunately, fidelity checks were not completed.
According to the program's Standardized Program Evaluation Protocol, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions. For the primary service of individual counseling, fidelity monitoring is conducted by
licensed clinician in the review of therapy notes to verify provision of effective and appropriate
intervention in line with the treatment plan and review of trackers to ensure compliance with
frequency and duration.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no fidelity
monitorings completed. It is unknown if there are any corrective actions needed.
No fidelity monitoring was conducted, so no points may be awarded for this indicator.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified are addressed continue to be an issue, the next stage is
completing a written warning and placing the facilitator on a performance improvement plan. The
performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 2019 through March 2020 show
there were facilitator evaluations completed. The facilitator evaluations had an area which
addressed facilitator skill delivering evidence-based practices. All facilitators trained in Impact of
Crime had evaluations which specifically addressed their performance.
Pursuant to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 18 youth in the
sample, 44% (8 of 18) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 18 youth in the
sample, 39% (7 of 18) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 39%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 78% (14 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 39% (7 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 20 25 80%
Totals 56 85 66%
This SPEP report evaluates Impact of Crime, an intervention delivered at Oak Grove Academy.
18 youth completed the program during the review period. Of those youth, 17 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 17 total youth sampled, only
13 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 8
received at least the recommended weeks of service. Youth in the sample completed between 7
and 24 weeks of service, with an average of 18 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, only 13 included dosage in the EBS Module. Of those youth with correct dosage, 7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
received at least the recommended hours of service. Youth in the sample completed between 8
and 44 hours of service, with an average of 26 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 72%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and by ensuring any corrective
action needed is identified, applied, and demonstrated.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Impact of Crime


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc.
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Seeking Safety
SPEP Review Date(s): 6/1/2020
Florida Department of Juvenile Justice
Report Date(s): 6/2/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 2019 through March 2020 shows
there are five staff trained in Seeking Safety. One therapeutic manager began running groups in
August 2019 but was not trained in the primary service until 10/4/2019.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services. Training for individual counseling services during On the
Job Training will be provided by a licensed clinician consisting of a review of common issues such
as trauma, mental health, substance abuse, violence victimization, impulsivity, and relationships
prevalent in the population being served. In addition, during On the Job Training, the procedures
for conducting the individual sessions including location, scheduling, and safety and security
concerns are covered.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the curriculum Seeking Safety: A Treatment Manual for PTSD and
Substance Abuse. The manual is broken down into twenty-five sessions. Each session explains
the topic the therapist will cover and provides handouts for the youth to complete. The primary
service can be utilized for individual treatment or group treatment.
The program used the incorrect sign in sheets during their first cohort beginning in May 2019. The
program identified the issue and began using the correct sign in sheets in July 2019.
Pursuant to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seeking Safety was unable to be observed due to the COVID-19 pandemic;
therefore, this indicator is rated as non-applicable.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did have staff turnover. One therapeutic manager was running groups from May
2019 through August 2019 but was not running the groups as prescribed; the clinical director
discontinued that group in the middle of the curriculum. That therapeutic manager is no longer
running Seeking Safety groups. There are currently three therapeutic managers running groups.
The documentation of Seeking Safety does not reveal any noticeable gaps in service. The
program opened in March 2019. Seeking Safety groups began on 5/7/2019 when the program
had enough youth to fill a group.
According to the program's Standardized Program Evaluation Protocol procedure, Oak Grove
Academy utilizes a probationary review status process designed for effective management of risk
though individualized coaching, training, and verification activities of all new employees during
their first ninety days. The process encourages personal growth and development, recognizes
areas of improvement, thus reducing staff turnover. In addition, the program provides positive
incentives, which in our experience has shown to improve employee motivation and dedication
and thus leads to better employee retention. Through this policy, employees receive credit for
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
years of service, employment referrals, and for contributing to the positive culture, teamwork
environment and for going above and beyond normal job expectations. Each employee who
receives credit for doing these things is entered in a monthly drawing at each facility. In the event
of facilitator turnover, the co-facilitator will assume the role of the facilitator for the group. If a co-
facilitator is not present, a new qualified facilitator for the primary service will take over conducting
the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of April 2019 through March 2020 show
there were nine internal fidelity monitorings completed. Those fidelity monitorings were completed
in July and October 2019, and January-February 2020.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the nine completed fidelity checks. The clinical director noticed discrepancies in
fidelity outside of regular monitoring and took appropriate action to correct the problem.
According to the program's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
the fidelity monitoring in a note and placed in the clinical supervision file. If during additional
fidelity monitoring the previously identified are addressed continue to be an issue, the next stage
is completing a written warning and placing the facilitator on a performance improvement plan.
The performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation for the annual review period of April 2019 through March 2020 show
there were facilitator evaluations completed. The facilitator evaluations had an area which
addressed facilitator skill delivering evidence-based practices. All facilitators trained in Seeking
Safety had evaluations which specifically addressed their performance.
Pursuant to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 39%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 78% (14 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 39% (7 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 20 25 80%
Totals 60 95 63%
This SPEP report evaluates Seeking Safety, an intervention delivered at Oak Grove Academy.
18 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. There were no end dates entered
into the Juvenile Justice Information system (JJIS) Evidence-Based Services (EBS) module for
any youths. With no JJIS data to review, the SPEP score was reduced. The program must enter
dosage data into JJIS in order to receive any score in this area.
The program earned 0 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, all 18 included dosage in the EBS Module. Of those youth, 0 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 2 and 32 hours of
service, with an average of 20 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 68%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring all
facilitators receive formal training prior to delivering services to youth; by ensuring facilitator
turnover does not impact groups; and by ensuring fidelity monitoring is completed every month for
each facilitator.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Seeking Safety


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Oak Grove Academy
Rite of Passage, Inc
(Contract Provider)
11180 NE 38th Street
Jasper, Florida 32052
Primary Service: Aggression Replacement Training
SPEP Review Period: April 1, 2020-March 31, 2021
SPEP Review Conducted: April 5, 2021
Florida Department of Juvenile Justice
Report Date: 5/5/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of April 1, 2020 through March 31, 2021 shows
the program utilizes Aggression Replacement Training as their primary service. Aggression
Replacement Training is provided by a qualified trainer over a three-day period. The program has
three staff who were running groups during the annual review period. Each facilitator also takes
turns conducting the monthly internal fidelity monitoring.
According to the program’s Standardized Program Evaluation Protocol procedure, the facilitator
for each primary service will have completed a formalized training for that primary service from a
qualified trainer. In order to enhance the continuity of the service due to unexpected facilitator
absence and provide more individualized instruction as needed during the service, when possible,
Oak Grove Academy will make all efforts to have a co-facilitator for the primary service. The co-
facilitator will have completed a formalized training for the primary service from a qualified trainer.
For a primary service without a formalized training, the facilitator will receive training from a
licensed clinician. This training will be documented in the facilitator’s On the Job Training Plan.
The facilitator will receive additional training on the curriculum as needed from the licensed
clinician documented within clinical supervision. All facilitators of a primary service will receive
training prior to conducting the services.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training (ART) manual. The manual is broken
down into six sections. The first section covers Skillstreaming. The second section deals with
anger control. The third section covers moral reasoning. Each of the first three sections has a
script for the facilitator to use for each lesson. Sections four through six are for the facilitator's
use. It includes training step guides, fidelity forms and hassle logs, and the program materials.
The program provides the students with copies of handouts for each lesson. The program
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
conducts the primary service on Wednesdays, Saturdays, and Sundays. The program keeps their
documentation for the primary service in a SPEP binder labeled ART. The binder contains all
attendance sheets, internal fidelity monitoring checklists, and group schedules. The training
certificates are held by the training coordinator at the program. All youth who are admitted to the
program receive the primary service. The primary service is held as a closed group.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Aggression Replacement Training was observed on 3/25/2021 via FaceTime. The
group began at 7p. The facilitator went over group rules. The lesson for the day was Week 6:
Keeping Out of Fights. The facilitator reviewed the skill step with the youth and then conducted
the role play to demonstrate the skill. The facilitator asked the youth if each skill step was a
thinking step or an action step. The facilitator asked each youth what their example was for
keeping out of fights. The facilitator then assigned the homework for the next class. The group
ended at 7:50p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program did not have facilitator turnover for Aggression
Replacement Training. The documentation showed there were three staff who facilitated the
primary service during the annual review period. There was one cohort completed and one cohort
which is currently running. The dates of the cohorts are 8/8/2020-11/1/2020 and 2/18/2021-
current. The primary service is a closed group run year-round, with the exception of the current
year due to the COVID-19 pandemic. The cohort which began on 8/8/2020 started with ten youth
and ended with seven youth. The cohort which began on 2/18/2021 started with nine youth.
There was a gap in service delivery of four months between the cohort which ended on 3/28/2020
(prior to this annual review period) and 8/8/2020. There was another gap in service delivery
between 11/1/2020 and 2/18/2021. The clinical director stated the program had to keep the youth
separated in their cottages due to the COVID-19 pandemic. The clinical director also stated there
was not enough youth in each cottage to begin a cohort. The cottages would have to be combined
in order to reach the minimum number of youth to begin a cohort. This service is not required to
be delivered to all youth, and each cohort was completed without gaps during services. These
gaps are not considered unneccessay.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the program's Standardized Program Evaluation Protocol, Oak Grove Academy
utilizes a probationary review status process designed for effective management of risk though
individualized coaching, training, and verification activities of all new employees during their first
ninety days. The process encourages personal growth and development, recognizes areas of
improvement, thus reducing staff turnover. In addition, the program provides positive incentives,
to improve employee motivation and dedication and lead to better employee retention. Through
this policy, employees receive credit for years of service, employment referrals, and for
contributing to the positive culture, teamwork environment and for going above and beyond
normal job expectations. Each employee who receives credit for doing these things is entered in a
monthly drawing at each facility. In the event of facilitator turnover, the co-facilitator will assume
the role of the facilitator for the group. If a co-facilitator is not present, a new qualified facilitator for
the primary service will take over conducting the service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed shows there were six Seeking Safety internal fidelity monitorings
completed over the course of three months. The program did not have any internal fidelity checks
completed during the months of February and March 2021. All trained staff perform monthly
internal fidelity checks.
According to the program's Standardized Program Evaluation Protocol procedure, designated
staff are responsible for conducting fidelity monitoring of each facilitator of a primary service on a
monthly basis. The staff conducting the fidelity monitoring has received formal training by a
qualified trainer in the primary service. Fidelity monitoring is documented on fidelity monitoring
tool developed specifically for that primary service by the developer of the manual, or and internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary services is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed showed there were no corrective actions noted on the four
completed monthly internal fidelity checks.
According to the program's Standardized Program Evaluation Protocol, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified are addressed continue to be an issue, the next stage is
completing a written warning and placing the facilitator on a performance improvement plan. The
performance improvement plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issues, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring and facilitator’s supervisor.
In the event that an issue identified during the primary service is detrimental to the youth’s safety
and /or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Annual evaluations for facilitators of Aggression Replacement Training (ART) were not provided.
Addenda were provided for review for four Therapeutic Managers; each was a copy/paste and did
not specifically address their skill in delivering ART. Additionally, all addenda were signed after
the end of the review period and the reviewer's initial request for documentation.
According to the program’s Standardized Program Evaluation Protocol procedure, an annual
evaluation of performance will be conducted on all the facilitators of each primary service. The
performance evaluation is comprised of feedback from fidelity monitoring tools as well as an
assessment of the facilitator’s skill delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 10 weeks for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 30 contact hours for this type of service. Of the 20 youth in the
sample, 70% (14 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 7 youth
High = 5 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 25%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 5 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (17 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 25% (5 of 20) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 18 25 40%
Totals 79 100 79%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Oak
Grove Academy.
36 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Duration. Of the 20 total youth sampled, only
3 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 6
and 8 weeks of service, with an average of 7 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, all 20 included dosage in the EBS Module. Of those youth, 14 received at least the
recommended hours of service. Youth in the sample completed between 6 and 30 hours of
service, with an average of 25 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Oak Grove Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator, and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Oak Grove Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10