summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Moral Reconation Therapy (MRT)
SPEP Review Date(s): October 25‐28, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/7/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: DOVE Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 24
Review Date(s): October 25-28, 2016 Lead Reviewer Code: 144
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff 1 # Other (listed by title): Boys &
DMHCA or designee 2 # Healthcare Staff Girls Club Staff
Documents Reviewed
Written Protocol/Manual Logbooks 11 # Personnel Records
Fidelity Monitoring Documents Program Schedules 13 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 4 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 10 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy (MRT). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The DOVE Academy has three staff certified to deliver Moral Reconation Therapy (MRT); clinical
director, clinical coordinator and a case manager. Each of these staff have completed formalized
training by the Correctional Counseling, Inc. All three of the MRT facilitators have conducted
training within the past twelve months.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The DOVE Academy facilitators of Moral Reconation Therapy (MRT), utilize the brand-name
manual entitled, “How to Escape Your Prison”. This is a curriculum specific for MRT, which was
written and co-authored by Dr. Gregory L. Little and Dr. Kenneth D. Robinson. The MRT facilitator
utilizes a workbook for groups and each youth receives a student workbook.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The DOVE Academy has had no turnover in Moral Reconation Therapy (MRT) facilitators. The
program has another trained MRT facilitator to take over classes if necessary.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The DOVE Academy Clinical Director will provide oversight and conduct monthly fidelity
monitoring for Moral Reconation Therapy (MRT). The program utilizes a fidelity monitoring tool
which complements the monitoring of MRT. The monitoring for MRT should be conducted
monthly, however, monitoring was missed for two months; May and June of 2016.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The DOVE Academy has a fidelity monitoring tool which contains key components to be
monitored during a Moral Reconation Therapy (MRT) group. The program’s process for
addressing any issues is to first consult with the clinical director. The next month the clinical
director would follow-up with any corrective action; training / re-training based upon findings from
the fidelity monitoring. There was no corrective action required during this review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The DOVE Academy’s annual performance evaluation addresses specific information concerning
the facilitator’s proficiency in the delivery of Moral Reconation Therapy (MRT).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 15 weeks for this type of service. Of the 9 youth in the
sample, 100% (9 of 9) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 3 youth
High = 4 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (9 of 9) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 44% (4 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 90 100 90%
This SPEP report evaluates Moral Reconation Therapy (MRT), an intervention delivered at DOVE
Academy.
The program scored High for Quality of Service Delivery. This score can be optimized by
ensuring that internal fidelity monitoring is conducted at least monthly.
The program earned 10 points for Amount of Service: Duration. Of the 9 total youth sampled, all
received at least the recommended weeks of service. Youth in the sample completed between 21
and 32 weeks of service, with an average of 25 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 9 total youth sampled,
none received at least the recommended hours of service. Youth in the sample completed
between 10 and 44 hours of service, with an average of 26 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Academy can optimize their SPEP Quality of Service Delivery score by ensuring that
internal fidelity monitoring is conducted at least monthly.
DOVE Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: DOVE Academy

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Seeking Safety
SPEP Review Date(s): October 25‐28, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/7/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: DOVE Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 24
Review Date(s): October 25-28, 2016 Lead Reviewer Code: 144
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff 1 # Other (listed by title): Boys &
DMHCA or designee 2 # Healthcare Staff Girls Club Staff
Documents Reviewed
Written Protocol/Manual Logbooks 11 # Personnel Records
Fidelity Monitoring Documents Program Schedules 13 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 4 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 10 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The DOVE Academy has three staff certified to deliver Seeking Safety curriculum; clinical director,
and two clinical coordinators. Each of these staff have completed training. Training is completed
by following the “Teaching Guide to Introduce Seeking Safety to Your Agency” and “Teaching
Guide to Introduce a Woman’s Path to Recovery”. All three of the Seeking Safety facilitators have
conducted training within the past twelve months.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The DOVE Academy facilitators of Seeking Safety, utilize the brand name manual entitled
Seeking Safety: A Treatment Manual for Post-Traumatic Stress Disorder (PTSD) and Substance
Abuse. The Seeking Safety curriculum is broken down into specific lesson plans and activity
worksheets for the facilitator and youth. A Seeking Safety group contains an introduction, group
discussion activity, outline, worksheets, and a conclusion/wrap-up.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The DOVE Academy conducted a Seeking Safety group on October 27, 2016. According to the
programs activity/group schedule, the Seeking Safety group observed started as scheduled. The
facilitator for the Seeking Safety group provided the observer with a copy of the lesson plan to
follow. The session conducted was well organized and youth were attentive to the facilitator and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
other youth participating within the group. The facilitator took the necessary time to explain the
objectives of the Seeking Safety session. The facilitator followed the manual as written.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The DOVE Academy has had one Seeking Safety facilitator leave within the past twelve months.
The clinical director was able to conduct the Seeking Safety group until a new therapist was
trained. The clinical director providing back-up group coverage has been trained with the Seeking
Safety curriculum. There was no indication of any lapse in service delivery for this curriculum.
The program can earn 2 points by minimizing facilitator turnover.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The DOVE Academy clinical director will provide oversight and conduct monthly fidelity
monitoring. The program utilizes a fidelity monitoring tool which compliments the monitoring of
Seeking Safety curriculum. The monitoring for Seeking Safety should be conducted monthly,
however, monitoring was missed for three months; May, June, and July of 2016. All three of the
Seeking Safety facilitators have conducted training within the past twelve months.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The DOVE Academy has a fidelity monitoring tool which contains key components to be
monitored during a Seeking Safety group. The program’s process for addressing any issues is to
first consult with the clinical director. The next month the clinical director would follow-up with any
corrective action; training/re-training based upon findings from the fidelity monitoring. There was
no corrective action required during this review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The DOVE Academy’s annual performance evaluation addresses specific information concerning
the facilitator’s proficiency in the delivery of Seeking Safety curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 47% (9 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 89% (17 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 87 95 92%
This SPEP report evaluates Seeking Safety, an intervention delivered at DOVE Academy.
The program scored High for Quality of Service Delivery. This score can be optimized by
minimizing facilitator turnover and by ensuring that internal fidelity monitoring is conducted at least
monthly.
The program earned 4 points for Amount of Service: Duration. Of the 19 total youth sampled, 9
received at least the recommended weeks of service. Youth in the sample completed between 11
and 39 weeks of service, with an average of 24 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, 17 received at least the recommended hours of service. Youth in the sample completed
between 21 and 74 hours of service, with an average of 46 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Academy can optimize their SPEP Quality of Service Delivery score by minimizing
facilitator turnover and by ensuring that internal fidelity monitoring is conducted at least monthly.
DOVE Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Dove Academy

program_status_note: Open

service_type: Seeking Safety


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Dove Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: The Seven Challenges
SPEP Review Date(s): October 24 ‐ 27, 2017
Florida Department of Juvenile Justice
Report Date(s): 05/22/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Dove Vocational Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 38
Review Date(s): October 24 - 27, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 9 # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 3 # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation reflected four current staff are trained in The Seven Challenges
curriculum. The designated mental health clinical authority (DMHCA) was trained in April 2013.
The DMHCA was then trained as a trainer of the curriculum in July 2013. The contracted therapist
was trained in April 2014. Two additional therapists were trained in June 2016. The facility is down
one therapist, however they have a therapist on contract that will come in to help cover groups,
treatment teams, and caseloads until the position is filled.
It was program protocol to have all therapists trained for substance abuse curricula and mental
health curricula based upon which group of youth the therapist will be working with. The DMHCA
is certified to train The Seven Challenges to staff.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The facility provided three manuals for The Seven Challenges curriculum. The facilitator's manual
was detailed and had 13 chapters, which covered implementation, the seven challenges,
counseling approaches, gender issues, supervision, games for groups, two appendices, four extra
worksheets, group note examples, and treatment goal examples. The curriculum is designed to
be run as an open group, meaning the youth and enter and complete group at their own pace.
The curriculum allows the therapist to be creative in how they want to facilitate the group. The
Seven Challenges is designed to be an open discussion for all youth to participate and apply the
lesson to their lives. The second manual provided was an activity book. This is the first program to
provide this manual. The activity book provides various counseling activities for each challenge.
Each activity provides materials, instruction, script, how to connect the lesson, and worksheets for
each lesson. The last manual was the journaling book the youth use during journaling sessions.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program created attendance sheets for this curriculum. Each attendance sheet had the
following sections: date, time, location of group, youth signature, youth participation (scored 0-2),
response to treatment, youth absent from group and reason, discussion notes, therapist name
printed and signature, session topic.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Seven Challenges group was observed on Tuesday, October 24, 2017. There were nine
youth who attended group out of 12 registered for the class. Three of the youth were working with
Department of Transportation on this date. This group was held in the library, which is a nice size
room. The topic worksheet was "Chasing Our Feelings and Forgetting Our Problems."
The facilitator maintained the group and kept the youth on topic. The facilitator maintained
objectivity and consistently used affirmations, reflections, and summaries throughout the group
time. The youth appeared to feel safe while in this group and have a good rapport with the
facilitator.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The attendance sheets for this group were accounted for and indicated no gaps in service. This
group is an open group and is continuously held. All of the attendance sheets were reviewed were
completely filled out 100% of the time.
During the review period one of the therapists left on medical leave, but later returned as a
contracted employee. This therapist is able to come and fill in as needed. When this therapist left
on maternity leave, another therapist covered her groups.
Currently, there are four therapists trained in The Seven Challenges curriculum and are able to
cover as needed to avoid gaps in service.
Two therapists are currently running two separate groups at this time for this curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring forms are all kept for all curricula in a binder labled "Mental Health and
Substance Abuse Fidelty Documents." There was a form located in the binder titled, "Fidelity
Monitoring - Evidenced Based Treatment Models," which included the process and procedure for
internal fidelity monitoring. Each curriculum was separated by a divider with the curriculum name
on the tab. The designated mental health clinical authority (DMCHA) conducts all internal fidelity
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
monitoring for The Seven Challenges. The forms provided were created by the facility. They were
detailed and included the following information: Program name, evidence-based model,
facilitator/clinical coordinator name, date, and reviewer. There are 11 questions to describe
feedback from observation of group. There is also a space for the reviewer signature and date.
There was a separate section for follow-up regarding any issues indicated. There were no missing
fidelity forms.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program director and designated mental health clinical authority (DMHCA) explained the
corrective action process during the review. If a corrective action was indicated on an internal
fidelity form, the supervisor would meet with the facilitator one-to-one to discuss. This would be
followed up with another observation to ensure a correction has occurred. For the therapists, they
meet with the DMHCA on a weekly basis.
There were no corrective actions indicated on the internal fidelity monitorings reviewed.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Three personnel files were provided for review by the program. All three of the therapists have
been employed at the program during the review period and had received a yearly performance
evaluation. The performance evaluations for therapists included a section titled, "Therapeutic
skills: delivery of delinquency interventions, gender specific, moral reconation therapy, The Seven
Challenges."
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 25% (5 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 4 youth
High = 9 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (18 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (9 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 75 95 79%
This SPEP report evaluates Seven Challenges, an intervention delivered at DOVE Vocational
Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 20 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 5
received at least the recommended weeks of service. Youth in the sample completed between 19
and 29 weeks of service, with an average of 24 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, only 17 included dosage in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended hours of service. Youth in the sample completed between 1
and 36 hours of service, with an average of 10 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
DOVE Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Seven Challenges


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Dove Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Impact of Crime
SPEP Review Date(s): October 24 ‐ 27, 2017
Florida Department of Juvenile Justice
Report Date(s): 05/22/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Dove Vocational Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development, Inc Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 38
Review Date(s): October 24 - 27, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 4 # Case Managers 8 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 7 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
During the review period the program employed four case managers who were all trained in the
Impact of Crime (IOC) curriculum. Documentation reflected that all four of the staff facilitated IOC
groups within the last year. The program's training protocol stated all case management staff
were to be trained to facilitate IOC.
The program has an ongoing plan to send all new case management staff to IOC training as often
as possible.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintained the most up-to-date Impact of Crime (IOC) facilitator's manual; there
were four sections to the IOC manual. The manual is very detailed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
During the review, an Impact of Crime group with eight youth was observed on Tuesday, October
24, 2017 from 3:15PM to 4:15PM. Twelve youth were registered for the class, however four of the
youth were working with Department of Transportation at the time of the session. The group was
held in the library located at the program. The group was on the third objective in the fourth
chapter. There was no co-facilitator present for this group. For this session, the group discussed
factors that interfered with good decision making. The youth were somewhat unruly, but the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
facilitator did a good job keeping control of the group. There were two staff that sat in group and
were helpful throughout the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program director wants all case managers to be trained in the IOC curriculum to assist with
any turnover and avoid gaps in service.
In the event of staff turnover, there are other case managers trained in the curriculum who would
be able facilitate the group to avoid any gaps in service.
There were four facilitators for the Impact of Crime groups during the review period. The facility
did not expereince any turnover nor gaps in delivery of service during this review period.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
During the review, four internal fidelity forms were reviewed. Internal fidelity monitoring
requirements for the IOC curriculum were discussed with the program director. It was suggested
one of the case managers could assist the program director in completing internal fidelity
monitoring for the other two facilitators in order to have them completed on a monthly basis.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The corrective action process was included in the fidelity monitoring completed by the program
director, who was also an Impact of Crime facilitator.
Of the four internal fidelity monitoring forms provided for review, no corrective actions were
indicated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Annual case management performance evaluations were provided for the case management
staff. All case management staff received a performance evaluation during the review period.
On the annual case management performance evaluations, there were no specific sections that
indicated group facilitation. There were no corrective actions on the yearly performance
evaluations.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 18 youth in the
sample, 22% (4 of 18) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 18 youth in the
sample, 22% (4 of 18) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 22 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 88%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 7 youth
Moderate-High = 5 youth
High = 6 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 6 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (18 of 18) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 33% (6 of 18) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 22 25 88%
Totals 66 85 78%
This SPEP report evaluates Impact of Crime, an intervention delivered at DOVE Vocational
Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 18 total youth sampled, only
4 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 4
received at least the recommended weeks of service. Youth in the sample completed an average
of 16 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, all included dosage in the EBS Module. Of those youth with correct dosage, 4 received
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
at least the recommended hours of service. Youth in the sample completed between 2 and 29
hours of service, with an average of 11 hours.
The program was awarded 22 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
DOVE Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Impact of Crime


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Dove Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Moral Reconation Therapy
SPEP Review Date(s): October 24 ‐ 27, 2017
Florida Department of Juvenile Justice
Report Date(s): 05/22/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Dove Vocational Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 38
Review Date(s): October 24 - 27, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 8 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 3 # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy. The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The facility provided the training records for all of the therapy staff in Moral Reconation Therapy
(MRT). One therapist was trained in May of 2010, a second therapist was trained in September of
2011, and a third therapist was trained in December of 2016. The designated mental health
clinical authority (DMHCA) was trained in MRT in May of 2010. The DMHCA was also trained as a
trainer for MRT in February of 2012.
The program's training protocol stated the DMHCA was in charge of training all new therapy staff
as they are hired at the facility. This is a formal 32-hour training for therapy staff.
At the time of this review, there were three ongoing Moral Reconation Therapy groups being
facilitated by three different therapists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided a workbook titled, "Juvenile MRT: How to Escape Your Prison, A Moral
Reconation Therapy Workbook." The book was written by Dr. Gregory Little and Dr. Kenneth
Robinson. For this curriculum, each youth is provided a workbook to work in throughout their time
in group. The program consisted of 12 steps that each youth must work through and successfully
complete in order to successfully complete the curriculum.
The manual did not provide very much detail, however all instructions for the youth were located
in their workbook. The therapist also used this workbook as a reference for the group. To
advance to each level, each youth must complete specific tasks and present to the group. The
group votes on advancement.
c. Observed Adherence to the Manual/Protocol Basic Score: 1 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A Moral Reconation Therapy group was observed on Wednesday, October 25, 2017 from
10:30AM to 11:30AM.
Throughout the session, the facilitator maintained control over the group. The facilitator had a
good rapport with each of the youth. The group was held with fidelity to the model.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
All of the attendance sheets for the MRT groups conducted in the last 12 months were reviewed.
All groups were accounted for and no gaps in service were noted. MRT is held as an open group,
meaning youth can start and complete group on their own pace. All of the attendance sheets were
filled out completely, with signatures. If a youth did not attend group, it was clearly noted on the
attendance sheets. The only suggestion would be to indicate when a youth had successfully
completed MRT on the attendance sheet.
In the event of staff turnover, an on-call therapist, who is contracted with the facility, would come
in and assist when needed. The program's protocol was for all therapists to be trained in MRT to
avoid any gaps in service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring forms were provided for review. They were all kept for all curricula in a
binder labled "Mental Health and Substance Abuse Fidelty Documents." There was a form located
in the binder titled, "Fidelity Monitoring - Evidenced Based Treatment Models", which included the
process and procedure for internal fidelity monitoring. Each curriculum is separated by a divider
with the curriculum name on the tab. The designated mental health clinical authority (DMCHA)
conducted all internal fidelity monitoring for Moral Reconation Therapy.
All internal fidelity monitoring had been conducted during the review period of 12 months.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program does have a process for corrective actions. There were no corrective actions for
either therapist indicated on the fidelity monitoring forms.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
All three of the therapists who facilitated MRT had received an annual performance evaluation.
The performance evaluations for therapists included a section titled, "Therapeutic skills: delivery
of delinquency interventions, gender specific, moral reconation therapy, The Seven Challenges."
There were no concerns or corrective actions indicated on the yearly performance evaluations.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 7 youth in the
sample, 14% (1 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 7 youth in the
sample, 0% (0 of 7) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 12 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 48%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 3 youth
High = 1 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 14%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 1 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (7 of 7) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 14% (1 of 7) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 12 25 48%
Totals 67 100 67%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at DOVE
Vocational Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 7 total youth sampled, only
1 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 1
received at least the recommended weeks of service. Youth in the sample completed an average
of 23 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 7 total youth sampled,
only 4 included dosage in the EBS Module. Of those youth with correct dosage, 0 received at
least the recommended hours of service. Youth in the sample completed between 10 and 20
hours of service, with an average of 15.25 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 12 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
DOVE Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Moral Reconation Therapy


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Dove Vocational Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Seeking Safety
SPEP Review Date(s): October 24 ‐ 27, 2017
Florida Department of Juvenile Justice
Report Date(s): 05/22/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Dove Vocational Academy MQI Program Code: 1102
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2113
Location: Jackson County / Circuit: 14 Number of Beds: 38
Review Date(s): October 24 - 27, 2017 Lead Reviewer Code: 151
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 12 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 3 # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seeking Safety. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program provided training records for all of the therapy staff. The designated mental health
clinical authority (DMHCA) was interviewed in regards to the training process for the Seeking
Safety curriculum. The training is conducted via video and there is not a set number of hours for
this training. It is considered informal and does not require a formalized training. The goal is to
have all therapuetic staff trained in the curriculum to be able to step in to continue a group in the
event of a therapist being out.
There are four theraputic staff trained in the Seeking Safety curriculum. All training certificates
were provided for the staff. All four therapuetic staff, including the DMHCA have facilitated the
Seeking Safety curriculum during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The designated mental health clinical authority (DMHCA) provided the Seeking Safety manual for
review. The manual is titled: Seeking Safety, Post Traumatic Stress Disorder (PTSD) and
Substance Abuse by Lisa Najavits. The treatment format is included in the manual. For this
curriculum the facilitator does not have to go in a particular order. There are no chapters, but
scenarios that can be chosen for the group to read and evaluate. The curriculum is 25 sessions,
but can be shorter if needed. The manual also provides a group formula, which included: group
check-ins, quotation of the day, handout discussion, youth are to relate the topics to their lives,
and check-out, which includes naming something that they learned during that lesson. The
manual also includes the following sections: Preparation for sessions description, session format,
session content, quotation, and handouts.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A Seeking Safety group was observed on Friday, October 27, 2017. This group is held seven
days a week and all youth in the program who are eligible to receive this curriculum. Twelve
youth were present for the group on this date.
The group was relaxed and inviting. The facilitator maintained the direction of the group, but was
not overbearing in how the group was held. The youth could speak freely and discuss openly.
The facilitator guided the discussion and asked questions. The youth did a good job of discussing
the topics and were engaged for each topic. The youth and facilitator used affirmations and
summaries during the group discussions.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There has not been facilitator turnover for the Seeking Safety currciulum. The program has
multiple therapists who delivery these groups in an open format. There were no gaps in service
during this review period.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
This reviewer spoke with the designated mental health clinical authority (DMHCA) regarding the
process for internal fidelilty monitoring. The DMHCA conducts all internal fidelity monitoring for all
curricula. Four therapists received internal fidelity monitoring during the review period. The
DMHCA started at the program in the spring of 2017 and was trained in the Seeking Safety
curriculum in May 2017.
The internal fidelity monitoring forms were provided. They are all kept for all curricula in a binder
labled "Mental Health and Substance Abuse Fidelty Documents." There is a form located in the
binder titled, "Fidelity Monitoring - Evidenced Based Treatment Models," which includes the
process and procedure for internal fidelity monitoring. Each curriculum is separated by a divider
with the curriculum name on the tab. The internal fidelity monitoring forms provided were created
by the facility. They are detailed and include the following information: Program name, evidence-
based model, facilitator/clinical coordinator name, date, and reviewer. The forms also include
feedback from observation of group and follow-up regarding any issues indicated.
It appears therapists were usually monitored for each month they facilitated group. The DMHCA
was on family leave during June and July and fidelity monitoring was not completed. The DMHCA
is the only other trained individual in the program, along with the therapists. It was suggested that
the therapists or a designated therapist could conduct the internal fidelity monitoring if the
DMHCA should need to leave again.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The designated mental health clinical authority (DMHCA) was interviewed in regards to the
process for correction actions with internal fidelity monitoring. The process is for the DMHCA to
meet one-on-one with each therapist during their weekly supervisions. If there is an issue during
group facilitation or if the therapist is on a corrective action it is discussed and followed up on a
regular basis until resolved.
After review of all fidelity monitoring forms, there were no corrective actions noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided yearly performance evaluations for therapeutic staff who have conducted
Seeking Safety in the last 12 months. The performance evaluations for therapists include a
section titled, "Therapeutic skills: delivery of delinquency interventions, gender specific, moral
reconation therapy, The Seven Challenges." There were no concerns or corrective actions
indicated on the yearly performance evaluations.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 26 youth in the
sample, 0% (0 of 26) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 26 youth in the
sample, 4% (1 of 26) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 6 youth
Moderate-High = 9 youth
High = 9 youth
Total Youth in Sample = 26 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 35%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 26 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (24 of 26) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 35% (9 of 26) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 73 95 77%
This SPEP report evaluates Seeking Safety, an intervention delivered at DOVE Vocational
Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 26 total youth sampled, only
1 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed an average
of 22 weeks. This average only includes youth with end dates in the EBS Module.
The program earned 0 points for Amount of Service: Contact Hours. Of the 26 total youth
sampled, all included dosage in the EBS Module. Of those youth with correct dosage, 1 received
at least the recommended hours of service. Youth in the sample completed between 1 and 58
hours of service, with an average of 13 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
DOVE Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
DOVE Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10