facility_name: Twin Oaks Academy II

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Academy II (TOVA II)
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 64 South
Sumatra, Florida 32335
Primary Service: LifeSkills Training (LST)
SPEP Review Date(s): December 6‐9, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/8/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Twin Oaks Vocational Academy II MQI Program Code: 1086
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2105
Location: Liberty County / Circuit: 2 Number of Beds: 12
Review Date(s): December 6-9, 2016 Lead Reviewer Code: 145
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff 1 # Other (listed by title): Clinical
DMHCA or designee # Healthcare Staff Director
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training (LST). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program currently has two clinical coordinators who are responsible for facilitating this
primary service. Documentation was provided indicating staff received training through the
Department of Juvenile Justice (DJJ) Technical Assistance staff. The program has completed one
full intervention cycle this review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the LifeSkills Training manual – Middle School Level edition, which is
appropriate for youth ages ten to thirteen years of age. Reviewer personally reviewed the manual
on-site during the annual quality improvement review.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The reviewer was unable to observe a session of the intervention during this review period.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program’s Clinical Director reported one clinical coordinator, responsible for facilitating the
primary service, resigned their position November 4, 2016. There was no gap in service delivery,
as the program continued to facilitate the groups with the other trained Clinical Coordinator on-
site. Currently, the program plans to send additional staff to be trained to facilitate the groups.
The program can earn 2 points by minimizing facilitator turnover.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Two facilitators provided service delivery this review period. There was no missing monitoring
documented for the past year. Documentation was provided to reveal fidelity monitoring
conducted during this annual compliance review period. Fidelity monitoring was conducted by
Department of Juvenile Justice (DJJ) Technical Assistance staff.
The program can earn 2 points by establishing their own internal fidelity monitoring process and
ensuring that fidelity monitoring is conducted at least monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program’s Clinical Director reports there is no in-house corrective action process for this
primary service. There was no corrective action indicated as needed this monitoring period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Copies of employee evaluations were provided for each staff who facilitated this primary service.
The evaluations contained to include specific information regarding the staffs’ delivery of the
primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 1 youth
High = 6 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 67%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (9 of 9) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 67% (6 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 65 85 76%
This SPEP report evaluates LifeSkills Training (LST), an intervention delivered at Twin Oaks
Academy II (TOVA II).
The program scored High for Quality of Service Delivery. This score can be optimized by
minimizing facilitator turnover, by establishing their own internal fidelity monitoring process, and
ensuring that fidelity monitoring is conducted at least monthly for all facilitators.
The program earned 0 points for Amount of Service: Duration and Amount of Service: Contact
Hours. There was no data entered into the Juvenile Justice Information system (JJIS) Evidence-
Based Services (EBS) module for any youths. With no JJIS data to review, the SPEP score was
reduced. The program must enter dosage data into JJIS in order to receive any score in this
area.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Twin Oaks Academy II (TOVA II) can optimize their SPEP Quality of Service Delivery score by
minimizing facilitator turnover, by establishing their own internal fidelity monitoring process, and
ensuring that fidelity monitoring is conducted at least monthly for all facilitators.
Twin Oaks Academy II (TOVA II) can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

program_name_raw: Twin Oaks Academy II

service_type: LifeSkills Training


state: FL

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: LifeSkills Training
SPEP Review Date(s): December 5 ‐ 8, 2017
Florida Department of Juvenile Justice
Report Date(s): 12/8/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Twin Oaks Vocational Academy II MQI Program Code: 1086
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2105
Location: Liberty County / Circuit: 2 Number of Beds: 32
Review Date(s): December 5 - 8, 2017 Lead Reviewer Code: 144
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 2 # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There are currently two clinical coordinators trained in LST curriculum. Training records were
provided for review. One clinical coordinator was formally trained July 2016 and the other clinical
coordinator was formally trained October 2017.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The designated mental health clinical authority (DMHCA) provided this reviewer with the LifeSkills
Training (LST) manual. The manual is the Middle School curriculum created by Botvin. The
manual includes three introductory sections: introduction, implementation guidelines, and
background information for the teacher. There are 15 chapters and a reference section.
Each chapter includes goals, objectives, materials needed, special prep, vocabulary, and
homework. The chapter guides the facilitator through the lesson. The manual does have a script
for the facilitator in each chapter. The program does not use the workbook associated with this
curriculum. Facilitators are encouraged to create worksheets and activities for the youth to
complete as part of the lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
LifeSkills Training (LST) group is held on Tuesdays and Thursdays from 9:45am ET to 10:45am
ET. Observation of the group was scheduled on Thursday morning during the review. However,
on that morning all but one of the youth from this group were out with the Department of
Transportation for the day.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Since this curriculum was not able to be observed; therefore, this indicator rates as non-
applicable.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was facilitator turnover during the review period. One clinical coordinator left the facility and
another was hired shortly after. This clinical coordinator was trained to conduct LST in October
2017. There were gaps in service from November 2016 to March 2017 and May 2017 until
October 2017. This reviewer spoke with the designated mental health clinical authority (DMHCA)
regarding the gaps in service. The youth committed to Twin Oaks Vocational Academy II have
been at a low census and the facilitators must wait until there are enough youth to conduct a
group.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Twin Oaks Juvenile Development, Inc created their own internal fidelity monitoring forms. The
forms are kept in a binder with all SPEP group information. They are detailed and include the
following information: Program name, evidence-based model, facilitator/clinical coordinator name,
date, and reviewer. There are 11 questions, to include feedback from observation of group. There
is also a space for the reviewer signature and date. There is a separate section for follow-up
regarding any issues indicated. Between December 2016 and October 2017 there was only one
clinical coordinator trained to conduct LST groups. During this time there was no other staff
trained in the facility that could conduct internal fidelity monitoring for this group.
Since October 2017 both clinical coordinators have received one internal fidelity monitoring each.
This reviewer met with the designated mental health authority (DMHCA) regarding the internal
fidelity monitorings. Now that there are two staff trained, each of them can conduct internal fidelity
monitoring on each other once a month moving forward.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a current process for internal fidelity monitoring. The two clinical coordinators
that are trained in Life Skills Training (LST) conduct internal fidelity monitoring on each other one
time a month. The clinical coordinators meet with the clinical director once per week for clinical
supervisions. If any corrective actions are noted on the internal fidelity forms, it is discussed at this
time. There were no corrective actions indicated on the two internal fidelity monitoring forms
completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Two staff facilitated LST during the review period. Both of the facilitators have received yearly
performance evaluations. There is one section on the yearly performance evaluation titled, "Job
Specific Duties". This section addressed group facilitation for all groups the employee was trained
in and currently facilitating. LST was listed as a curriculum on both of the yearly performance
evaluations.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 4 youth in the
sample, 0% (0 of 4) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 4 youth in the
sample, 0% (0 of 4) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 1 youth
High = 2 youth
Total Youth in Sample = 4 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 2 youth
Total Youth in Sample = 4 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (4 of 4) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 50% (2 of 4) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 55 85 65%
This SPEP report evaluates LifeSkills Training, an intervention delivered at Twin Oaks Vocational
Academy.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 4 total youth sampled, none
included dosage with end dates in the EBS Module.
The program earned 0 points for Amount of Service: Contact Hours. Of the 4 total youth sampled,
all 4 included dosage in the EBS Module. Of those youth, 0 received at least the recommended
hours of service. Youth in the sample completed between 4 and 6 hours of service, with an
average of 4 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Twin Oaks Vocational Academy can optimize their SPEP Quality of Service Delivery score by
ensuring there are no gaps in service due to facilitator turnover, and by ensuring fidelity
monitoring is conducted more frequently than during this review period.
Twin Oaks Vocational Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_status_note: Open


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: LifeSkills Training
SPEP Review Date(s): May 14‐17, 2019
Florida Department of Juvenile Justice
Report Date(s): May 17, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation provided by the program reflected the program has a training protocol for
LifeSkills Training. The program has one staff trained to facilitate the intervention; she was
trained by a qualified trainer in 2016.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Botvin LifeSkills Training (LST), Middle School program, curriculum. The
manual includes three introductory sections: introduction, implementation guidelines, and
background information for the facilitator. There are fifteen (15) chapters and a reference section.
Each chapter includes goals, objectives, materials needed, special prep, vocabulary, and
homework. The chapter guides the facilitator through the lesson. The manual has a script for the
facilitator in each chapter.
All youth in the Twin Oaks Vocational Academy II program receive LST. Groups are held
Tuesdays and Thursdays, from 2:00PM to 3:00PM.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The most recent cohort ended the week prior to the review. An observation was unable to be
conducted; therefore, this indicator is rated as not applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has a process in place to avoid gaps in service due to facilitator turnover. There
was no facilitator turnover during the review period. There was one gap in service from 04/18
until 11/4/18; however, this gap did not adversely affect services to youth as all youth received the
intervention during the review period.
The program currently has only one staff trained to facilitate LifeSkills Training. It was
recommended additional staff be trained to avoid any possible future gaps in service.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct internal fidelity monitoring for SPEP services;
however, there is only one facilitator employed at the program and therefore there is no one
available to conduct internal fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions resulting from internal fidelity
monitoring. The Clinical Director conducts weekly supervision meetings with all clinical staff. If an
issue were noted on the internal fidelity monitoring forms, it is discussed during these meetings
and followed up on during the next internal fidelity monitoring.
No fidelity monitoring was completed during the review period; therefore, no points may be
awarded in this section.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitator’s performance evaluation during the review period confirmed evaluation
of the staff’s performance in delivering LifeSkills Training. The evaluation included a goal for
improvement during the next review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 15 weeks for this type of service. Of the 11 youth in the
sample, 64% (7 of 11) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 15 contact hours for this type of service. Of the 11 youth in the
sample, 82% (9 of 11) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 5 youth
Total Youth in Sample = 11 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 11 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (10 of 11) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (5 of 11) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates LifeSkills Training, an intervention delivered at Twin Oaks Vocational
Academy.
14 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 15 weeks; the targeted number of
hours for this specific intervention is 15 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 3 services (16 weeks; 24
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 6 points for Amount of Service: Duration. Of the 11 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended weeks of service. Youth in the sample completed between 3
and 21 weeks of service, with an average of 17 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 11 total youth
sampled, all 11 included dosage in the EBS Module. Of those youth, 9 received at least the
recommended hours of service. Youth in the sample completed between 3 and 37 hours of
service, with an average of 27 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Twin Oaks Vocational Academy can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator.
Twin Oaks Vocational Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: LifeSkills Training
SPEP Review Date(s): 05/01/2020
Florida Department of Juvenile Justice
Report Date(s): 5/1/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program typically has the case coordinator or a clinical staff trained in LifeSkills Training. The
program currently has one formally trained staff member. The staff member was trained in 2016
by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Botvin LifeSkills Training (LST), Middle School program, curriculum. The
manual includes three introductory sections: introduction, implementation guidelines, and
background information for the facilitator. There are fifteen (15) chapters and a reference section.
Each chapter includes goals, objectives, materials needed, special prep, vocabulary, and
homework. The chapter guides the facilitator through the lesson. The manual has a script for the
facilitator in each chapter. All youth in the Twin Oaks Vocational Academy II program receive
LST. Groups are held Tuesdays and Thursdays from 1:00pm to 2:00pm.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The group was not being held during the quarterly review in October 2019. At this time the group
cannot be observed due to COVID-19 protocols. Once those protocols are lifted, this reviewer will
schedule a group observation. This indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There is currently one staff member trained in Life Skills Training (LST) curriculum. The program
has been unable to have other staff trained during this review period. There has not been staff
turnover during this review period. If there is turnover there is currently no other staff member to
continue LST groups.
One full cohort was held between October 15, 2019 ad December 19, 2019. The groups were
held for one hour on a regular basis. The group started with seven youth and ended with six
youth. One youth left the program on December 19, 2019 and did not successfully complete the
curriculum. LST groups were not held between May 2019 and October 2019; however, this is not
due to facilitator turnover, but rather a lack of new admissions to begin a new cohort. Between
May 2019 and October 2019 four new youth were admitted. These youth attended the group held
between October 2019 and December 2019.
A second cohort was started on April 9, 2020. Again, this gap was not due to facilitator turnover,
but due to a lack of new admissions to begin a new cohort. There are three youth in the current
cohort. Although this is below the recommended amount of youth to begin a group, these youth
needed to complete the group prior to graduating the program.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct internal fidelity monitoring for Standardized
Program Evaluation Protocol services; however, there is only one facilitator employed at the
program and therefore there is no one available to conduct internal fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was no internal fidelity monitoring completed during the review period, therefore no
corrective actions were noted.
Internal fidelity monitoring must be conducted regularly to receive a score in this section.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitator’s performance evaluation during the review period confirmed evaluation
of the staff’s performance in delivering LifeSkills Training.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 11 youth in the
sample, 91% (10 of 11) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 15 contact hours for this type of service. Of the 11 youth in the
sample, 91% (10 of 11) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 82%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 2 youth
Moderate-High = 1 youth
High = 6 youth
Total Youth in Sample = 11 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 55%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 11 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 82% (9 of 11) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 55% (6 of 11) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 66 85 78%
This SPEP report evaluates LifeSkills Training, an intervention delivered at Twin Oaks Vocational
Academy.
11 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks (for this program only);
the targeted number of hours for this specific intervention is 15 hours. After additional review and
discussion with the Peabody Research Institute at Vanderbilt University, the Department
determined it was within SPEP standards to deviate from the standard dosage targets for Group 3
services (16 weeks; 24 hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Duration. Of the 11 total youth sampled, all
11 included dosage with end dates in the EBS Module. Of those youth, 10 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 12 weeks of
service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 11 total youth
sampled, all 11 included dosage in the EBS Module. Of those youth, 10 received at least the
recommended hours of service. Youth in the sample completed between 11 and 20 hours of
service, with an average of 17 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 84%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Twin Oaks Vocational Academy can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator and by ensuring any
corrective action needed is identified, applied, and demonstrated.
Twin Oaks Vocational Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Vocational Academy II
Twin Oaks Juvenile Development, Inc
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: LifeSkills Training
SPEP Review Period: May 15, 2020 - May 14, 2021
SPEP Review Conducted: May 18, 2021 - May 21, 2021
Florida Department of Juvenile Justice
Report Date: 8/11/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes LifeSkills Training as a primary service.The program's training protocol is for
a licensed therapist or certified trainer to conduct LifeSkills Training groups.
The program's documentation reflected there were three trained staff during the twelve month
review period. All training documentation was provided for this review and all staff were formally
trained. Two staff were traned December 2020 and the third staff was trained in 2016. Two of the
three trained staff facilitated Life Skills Training groups during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Botvin LifeSkills Training (LST), Middle School program, curriculum. The
manual includes three introductory sections: introduction, implementation guidelines, and
background information for the facilitator. There are fifteen chapters and a reference section.
Each chapter includes goals, objectives, materials needed, special prep, vocabulary, and
homework. The chapter guides the facilitator through the lesson. The manual has a script for the
facilitator in each chapter.
All youth in the Twin Oaks Vocational Academy II program receive LST. The groups are currently
held on Mondays from 3:00pm to 4:00pm and Thursdays from 2:00pm to 3:00pm. A binder is
maintained in the clinical director's office with all documentation for LST groups.
Eligibility for the LST group is based upon the youth's criminogenic needs identified on the
Residential Assessment for Youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
LifeSkills Training was observed on March 1, 2021 at 3:00pm. Present for the group were five
youth, one direct care staff, the faciltator, the clinical director, and the observer. Youth were
discussing the lesson "How to be More Assertive." The facilitator followed the lesson plan, using
her own words to relay the message to youth. The facilitator worked through each section of the
material, verifying everyone grasped the concepts before moving on. At the end of the lesson, the
facilitator summarized the material and asked each youth to provide their own examples. Group
lasted approximately thirty minutes, in accordance with the curriculum design. The facilitator was
able to recognize the youth were no longer listening and unable to absorb more information with
the curriculum. Therefore the group ended at approximately thirty minutes.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced LifeSkills Training facilitator turnover, according to the clinical director.
One trained staff left during the review period; however, this did not disrupt group due to the
facilitator completing the group being held prior to leaving employment. Should the program
experience turnover during a cohort of LifeSkills Training, another trained facilitator would
continue with the group.
There were two gaps in service between July 2020 and October 2020 and March 2021 to May
2021 due to a low census in the program. This gap was not caused by staff turnover. The
program experienced a significant lack of youth entering the program during this timeframe. This
does not qualify as an unnecessary gap in service. All youth in the program were able to complete
this service.
The program completed three cohorts during the twelve month review period.
The first cohort was held from April 9, 2020 to July 27, 2020. Three youth started this group and
three youth successfully completed. The cohort had a low number of youth attending due to a low
amount of youth in the program. The program received permission from the Department of
Juvenile Justice to conduct the cohort with a lower amount of youth than is required.
The second cohort was held from October 8, 2020 to December 3, 2020. There were two youth
who started the group and two youth successfully completed. The cohort had a low number of
youth attending due to a low amount of youth in the program. The program received permission
from the Department of Juvenile Justice to conduct the cohort with a lower amount of youth than
is required.
The third cohort was held from January 4, 2021 to March 17, 2021. There were four youth who
started the group and four youth who successfully completed. The cohort had a low number of
youth attending due to a low amount of youth in the program. The program received permission
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
from the Department of Juvenile Justice to conduct the cohort with a lower amount of youth than
is required.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy for internal fidelity monitoring is, "Fidelity monitoring is conducted by trained
program personnel who have complete formal training in the evidence based model being
monitored. Corporate management staff assists in all fidelity monitoring reviews. In no case will
program staff monitor their own services provision. However, in accordance with Department
standards, it is allowable for staff in the same program to monitor each other."
The internal fidelity monitoring forms are thorough and include feedback from the observer, a
section for corrective actions, and a section for behavioral issues addressed during the group with
the youth. All internal fidelity monitoring forms included the faciltator's name, the observer's name,
and the date and time of the group being observed.
There were three facilitators of LifeSkills Training during this review period. There were no other
staff trained in the program to conduct internal fidelity monitoring for LifeSkills Training until
December of 2020. There were three internal fidelity monitorings conducted during the last
quarter of the twelve month review period. Three of three internal fidelity monitoring forms were
provided for review.
There were seven out of ten internal fidelity monitoring forms missing for the twelve month review
period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy for addressing corrective actions is "Corrective actions will be issued when
programs are not conducting treatment as prescribed or when other problems are identified during
the monitoring process. The Chief Operations Officer will be responsible for ensuring that
problems identified are corrected in a timely manner."
No corrective actions or issues were noted on the fidelity monitoring forms for LifeSkills Training.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the employee's annual performance evaluation found a section addressing the
evaluation of the facilitator in delivering LifeSkills Training. One of the two facilitators had received
an annual performance evaluation during this review period. The other facilitator left employment
before one was completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 10 youth in the
sample, 70% (7 of 10) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 15 contact hours for this type of service. Of the 10 youth in the
sample, 100% (10 of 10) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 10 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 3 youth
Total Youth in Sample = 10 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (9 of 10) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 30% (3 of 10) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 76 85 89%
This SPEP report evaluates LifeSkills Training, an intervention delivered at Twin Oaks Vocational
Academy II.
10 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks (for this program only);
the targeted number of hours for this specific intervention is 15 hours. After additional review and
discussion with the Peabody Research Institute at Vanderbilt University, the Department
determined it was within SPEP standards to deviate from the standard dosage targets for Group 3
services (16 weeks; 24 hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 6 points for Amount of Service: Duration. Of the 10 total youth sampled, all
10 included dosage with end dates in the EBS Module. Of those youth, 7 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 16 weeks of
service, with an average of 12 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 10 total youth
sampled, all 10 included dosage in the EBS Module. Of those youth, 10 received at least the
recommended hours of service. Youth in the sample completed between 16 and 29 hours of
service, with an average of 18 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 95%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Twin Oaks Vocational Academy II can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator.
Twin Oaks Vocational Academy II can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2020-21 TOVA LST.pdf


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Twin Oaks Vocational Academy II
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: LifeSkills Training
SPEP Review Period: November 1, 2021 - October 31, 2022
SPEP Review Conducted: December 6, 2022
Florida Department of Juvenile Justice
Report Date: 2/20/23
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is LifeSkills Training.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes LifeSkills Training (LST) as a primary service in the program. The therapy
staff are trained to facilitate LST groups. All training records were provided for review. There were
three trained staff during the annual review period. The staff were trained by the developers of
Botvin's LifeSkills Training. There was one main facilitator during the annual review period.
The trained facilitators monitor each other for internal observations on a monthly basis.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Botvin LifeSkills (LST) manual. The facilitators each have their own
manual and workbooks for the youth. The manual includes a script, lesson plans, and facilitator
guidelines.
LifeSkills Training is scheduled for Tuesdays and Thursdays from 1:00pm to 2:00pm. The
program maintains sign-in sheets and fidelity monitoring observations in a binder kept in the
clinical director's office. All youth are eligible to receive the LST curriculum. The groups are closed
groups and youth may not be added to the cohort after completion of the first lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A LifeSkills Training group was observed on Tuesday, December 13, 2022. There were two youth
and one facilitator. The group was on Part one of Chapter 3: Smoking: myth and reality. The youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
and facilitator reviewed definitions for the new chapter. While they went through the lesson, they
connected each myth versus reality back to the definitions. The facilitator did a good job rolling
with statements the youth make and connecting it back to the lesson. The group had great
conversations with the facilitator when going through the lesson. The facilitator had all items
prepared for group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol, in the event of facilitator turnover for LifeSkills Training (LST), is for
another trained staff to continue with the group facilitation. The program did experience turnover
during the annual review period. One staff was no longer facilitating due to information received
by the Office of Health Services. The program was informed a bachelor's level staff was ineligible
to deliver the clinical service, which was inaccurate information, pursuant to a review of the
contract. Due to this, there was one facilitator during the second and third quarters of the annual
review period.
There were two gaps in service during the annual review period: June 15, 2022 - August 8, 2022
(fifty-four [54] days) and October 28, 2022 to December 5, 2022 (thirty-seven [37] days). The
program completed three cohorts during the annual review period.
The first cohort started November 18, 2021 to February 9, 2022. The cohort started with three
youth and three youth successfully completed the group.
A second cohort was started on January 25, 2022 and was stopped on February 8, 2022 at the
direction of the Office of Health Services. The youth completed a subsequent cohort.
A third cohort started on March 28, 2022 and ended on June 15, 2022. The cohort started with
two youth and two youth successfully completed the cohort. This cohort was held multiple times
during the week instead of the scheduled one time a week for one hour.
A fourth cohort started on August 8, 2022 and ended on October 28, 2022. The group started with
two youth and three youth successfully completed the cohort. The program received permission to
add a youth to the cohort who was admitted to the program shortly after the cohort began. The
youth received individual sessions to be caught up and then joined the group.
A fifth cohort started on December 5, 2022 and was ongoing at the time of the annual review.
There were two youth in the cohort.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Trained staff conduct internal fidelity monitoring on each other for LifeSkills Training (LST) groups.
A review of documentation provided and a discussion with the Clinical Director revealed fidelity
monitoring was conducted each of the four months groups were held during the review period.
Fidelity monitoring was conducted for the bachelor’s level staff on 11/22/21, 12/31/21, 1/12/22,
and 2/7/22.
There was no one else trained to conduct internal fidelity monitoring between March and June of
2022. Internal fidelity observations picked back up in September 2022 when the other staff was
trained to facilitate LST groups.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any issues or corrective actions with the facilitators for LifeSkills
Training (LST). The internal fidelity monitoring forms used for LST groups include a section to
document any significant issues or corrective actions.
A review of the internal fidelity forms did not indicate there were any significant issues or
corrective actions during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual performance evaluations for the facilitators for LifeSkills Training
(LST). The evaulations include a section to document the curricula the staff are trained in.
At the time of the annual review only one trained staff member had been employed for one year
and this evaluation was completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 6 youth in the
sample, 33% (2 of 6) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 15 contact hours for this type of service. Of the 6 youth in the
sample, 83% (5 of 6) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 7 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 28%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 0 youth
Total Youth in Sample = 6 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 6 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (5 of 6) youth scored Moderate to High-Risk to reoffend, for a score of
7 points.
Of the SPEP sample, 0% (0 of 6) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 7 25 28%
Totals 57 85 67%
This SPEP report evaluates LifeSkills Training, an intervention delivered at Twin Oaks Vocational
Academy II.
6 youth completed the program during the review period. Of those youth, all 6 were included in
the sample.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 15 weeks; the targeted number of
hours for this specific intervention is 15 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 3 services (16 weeks; 24
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Duration. Of the 6 total youth sampled, only
5 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 2
received at least the recommended weeks of service. Youth in the sample completed between 12
and 23 weeks of service, with an average of 16 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 6 total youth sampled,
only 5 included dosage in the EBS Module. Of those youth with correct dosage, 5 received at
least the recommended hours of service. Youth in the sample completed between 20 and 25
hours of service, with an average of 23 hours.
The program was awarded 7 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Twin Oaks Vocational Academy II can optimize their SPEP Quality of Service Delivery score by
ensuring there are no unnecessary gaps in service and fidelity monitoring is completed every
month for each facilitator.
Twin Oaks Vocational Academy II can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 TOVAII LST.pdf