facility_name: Juvenile Unit for Specialized Treatment (JUST)

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Date(s): December 5 ‐ December 8, 2017
Florida Department of Juvenile Justice
Report Date(s): 6/8/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Juvenile Unit for Specialized Treatment MQI Program Code: 1086
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2105
Location: Liberty County / Circuit: 2 Number of Beds: 32
Review Date(s): December 5 - December 8, 2017 Lead Reviewer Code: 144
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 2 # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
During the review period, the program's documentation reflected that the program had a training
protocol and six staff members received formal training in the Impact of Crime (IOC) curriculum to
become a group facilitator. The six trained staff included two therapists, one formally trained in
January 2012 and the second in June 2017, two clinical coordinators, both trained June 2017, and
a case manager, who was formally trained June 2017. One direct care staff was also previously
trained, but had not conducted IOC during this review period. This reviewer discussed the plan to
have all trained staff facilitate a group with the clinical supervisor.
Only two of the six will continue conducting IOC groups. The two therapists were the only two staff
members who conducted IOC group during this review period and they will continue.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for reference during
the review.
The program maintained the current Impact of Crime (IOC) facilitator's manual. There were four
sections to the IOC manual and it is very detailed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A session of Impact of Crime (IOC) was observed on Thursday, December 7, 2017 from 2:40PM
to 3:40PM. A copy of the lesson plan was provided. The sessions are typically faciliated by two
therapists, however, one therapist was on leave during the review.
The group started with and icebreaker to get the youth engaged and talking. The facilitator then
had the youth sit down and introduced chapter 4. The facilitator provided feedback to the youth
and debriefed the role play by asking the youth questions from the script and allowed for
discussion. The facilitator consistently used open-ended questions, affirmations, and reflections.
The facilitator followed the lesson according to the manual, did not read the lesson verbatim, used
her own words to convey the meaning of the discussed topic, and did so easily.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Upon review of the facility documentation, there have been no gaps in service that adversly
affected the youth during the twelve month review period. There were two gaps in service during
the review period. The first gap was between March 2017 and May 2017 and the second between
August 2017 and October 2017. This reviewer spoke with the clinical director to inquire about the
gaps in service, since there had not been turnover in staffing. The clinical director explained that
there were not enough youth to begin a group until after other youth were finished with the other
evidence-based group. The other issue discussed by the clinical director was that the length of
stay at this program is short and does not allow enough time for the youth to complete both
groups. It was suggested that both groups are run concurrently in order to serve all youth in the
program.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's current internal fidelity monitoring practice is for each of the staff currently
facilitating Impact of Crime (IOC) groups to monitor each other once per month. All internal fidelity
monitoring is provided to the clinical director for review. If there are any issues noted, it is
discussed during weekly supervisions with the clinical director. It was suggested by this reviewer
to have a third, formally trained staff member to conduct internal fidelity monitoring. Twin Oaks
Juvenile Development, Inc created their own internal fidelity monitoring forms. The forms are kept
in a binder with all SPEP group information.They are detailed and include the following
information: Program name, evidence-based model, facilitator/clinical coordinator name, date, and
reviewer. There are 11 questions that get answered, to include feedback from observation of
group. There is also a space for the reviewer signature and date. There is a separate section for
follow-up regarding any issues indicated.
There were two internal fidelity monitoring forms reviewed for the past twelve months. There was
one internal fidelity form completed for each facilitator. One was completed October 2017 and the
other was completed November 2017. Each of the facilitators should have received internal
fidelity monitoring for October and November 2017.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Prior to June 2017, there was only one therapist trained to facilitate the IOC curriculum, so no
other internal fidelity monitoring was completed.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the clinical director verified that the program has a process for corrective
actions based on fidelity monitoring conducted. The internal forms created by Twin Oaks Juvenile
Development, Inc. has a section specific to corrective actions. Of the two forms reviewed, no
corrective actions were noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the two clinical coordinators' employee performance evaluations found an entry for
evaluation of the staff's performance in delivering Impact of Crime. The comments included
strenghts and weaknesses of which the facilitators should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 10 youth in the
sample, 0% (0 of 10) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 10 youth in the
sample, 20% (2 of 10) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 10 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 3 youth
Total Youth in Sample = 10 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (9 of 10) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 30% (3 of 10) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 20 25 80%
Totals 62 85 73%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 10 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between an
average of 11 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 10 total youth
sampled, only 2 included dosage in the EBS Module. Of those youth with correct dosage, 2
received at least the recommended hours of service. Youth in the sample completed between 28
and 29 hours of service, with an average of 28 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring youth receive
the full targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced dosage score
for this program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: Thinking for a Change
SPEP Review Date(s): December 5 ‐ 8, 2017
Florida Department of Juvenile Justice
Report Date(s): 6/8/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Juvenile Unit for Specialized Treatment MQI Program Code: 1086
Provider Name: Twin Oaks Juvenile Development, Inc. Contract Number: R2105
Location: Liberty County / Circuit: 2 Number of Beds: 32
Review Date(s): December 5 - 8, 2017 Lead Reviewer Code: 144
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation reflected the program had a training protocol and that three clinical
staff received formal training from a qualified trainer to become a group facilitator in Thinking for a
Change (T4C). The clinical director was trained as a facilitator in 2014 and became a qualified
trainer in 2017. Two clinical coordinators were trained in August 2017. Until August 2017, the
clinical director was the only staff member trained from December 2016 until August 2017.
Per a conversation with the clinical director, it was determined that these three staff will remain
the T4C facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for this reviewer to
reference during the review. The program currently uses Thinking for a Change (T4C) version 4.0.
The manual provides a script for each of the 25 lessons. The youth are introduced to the skill and
skill steps for each lesson. There are facilitator notes located in the margins of each page.
All skill cards, slides, and homework sheets are no located online. All three facilitators have
access to the materials needed to conduct T4C. The manual and supplemental materials are
detailed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A Thinking for a Change group was observed during the review on Tuesday, December 5, 2017.
Ten youth were registered for this group, however seven attended on this day. Three youth were
at doctor's appointments. The facilitators provided a copy of the lesson from the program manual.
The facilitators followed the lesson and appeared comfortable with the material. Both facilitators
were able to put the lesson into their own words and did not read the lesson verbatim.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation with the clinical director confirmed that there has been no turnover
that adervsersly affected the youth during the past twelve months. To avoid gaps in service, this
group will be co-facilitated, so in the event of turnover, there will be two other staff members that
are able to continue the group. There was a gap in service between December 2016 and March
2017. The clinical director stated that the facilitators had to wait until there were enough youth to
begin a group.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program had an internal process to conduct internal fidelity monitoring specific to Thinking for
a Change (T4C). This process is for the clinical director to conduct internal fidelity monitoring for
the two clinical coordinators. The clinical director is trained in T4C as a facilitator and a trainer.
Twin Oaks Juvenile Development, Inc created their own internal fidelity monitoring forms. The
forms are kept in a binder with all SPEP group information. The forms are detailed and include the
following information: Program name, evidence-based model, facilitator/clinical coordinator name,
date, and reviewer. There are 11 questions that get answered, to include feedback from
observation of group. There is also a space for the reviewer signature and date. There is a
separate section for follow-up regarding any issues indicated.
There were two internal fidelity monitoring forms provided to this reviewer. Prior to the clinical
coordinators being trained and beginning a group in October 2017, the clinical director was the
only trained staff member to facilitate T4C from December 2016 to June 2017, and fidelity
monitoring could not be completed.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the clinical director verified that the program has a process for corrective actions
based on fidelity monitoring conducted. The internal fidelity monitoring forms are completed by the
clinical director. The clinical director conducts weekly supervision meetings with all clinical staff. If
there was an issue indicated on the internal fidelity monitoring forms, it would be discussed during
weekly supervision and followed up with the next internal fidelity monitoring. The two internal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
fidelity monitoring forms this reviewer observed did not have any corrective actions or
recommendations.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of three facilitator's employee performance evaluations found an entry for evaluation of
the staff's performance in delivering Thinking for a Change. There were set goals regarding group
facilitation for the next review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 7 youth in the
sample, 86% (6 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 45 contact hours for this type of service. Of the 7 youth in the
sample, 5% (5 of 7) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 2 youth
High = 4 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (6 of 7) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 57% (4 of 7) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 92 100 92%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Juvenile Unit for
Specialized Treatment.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 7 total youth sampled, all 7
included dosage with end dates in the EBS Module. Of those youth, 6 received at least the
recommended weeks of service. Youth in the sample completed between 14 and 44 weeks of
service, with an average of 22 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 7 total youth sampled,
all 7 included dosage in the EBS Module. Of those youth, 5 received at least the recommended
hours of service. Youth in the sample completed between 28 and 45 hours of service, with an
average of 40 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery
score by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Juvenile Unit For Specialized Treatment

program_status_note: Open

service_type: Thinking for a Change


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Date(s): May 14‐17, 2019
Florida Department of Juvenile Justice
Report Date(s): May 17, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation provided by the program reflected the program has a training protocol for Impact
of Crime, and three (3) staff are trained in the curriculum. Those staff are the Clinical Director and
two Clinical Coordinators. Of the three staff, one facilitated group during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilizes the Impact of Crime, Addressing the Harm curriculum, version
2.0. The program has a manual for the delivery of services and a workbook for the students to
use. The curriculum provides a script and instructions for seven chapters which requires a
minimum of thirty-two contact hours to facilitate. The curriculum includes implementation
guidelines, identifies the order the lessons are to be delivered, and provides detailed instructions
for the delivery of each lesson.
Youth are placed in the intervention based on identified criminogenic needs based on the
Residential Positive Achievement Change Tool. Youth not placed in Impact of Crime are placed
in Thinking for a Change. Groups are held Mondays and Wednesdays, from 3:30PM until
4:30PM.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on Wednesday, May 15, 2019, from 3:30PM to
4:30PM. The group was discussing Chapter 3, Introduction to Harm, a lesson on understanding
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
insurance and deductibles. Ten (10) youth were present for group. The facilitator took care to
ensure youth understood the subject; she was patient, helpful, encouraging, and knowledgeable.
At the end of the session, the facilitator successfully wrapped up by circling back to the beginning
of the lesson, discussing the different types of harm. The facilitator was knowledgeable of the
material and followed the lesson plan without reading verbatim. The facilitator adhered to the
lesson plan in every aspect.
At two (2) separate points during the group, youth were brought in to be supervised by the direct
care staff present for group. The youths were immediately removed by the Clinical Director, who
was also observing the group. It was disclosed the youth were brought in to bring another staff
member elsewhere on campus back into ratio and was acknowledged to be an unacceptable
practice. The Clinical Director confirmed she will address and correct this practice with program
staff.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has a process in place to avoid gaps in service due to facilitator turnover. There
was no turnover during the review period. There was a gap in service from 8/8/18 through
3/31/19; however, this gap was due to there not being enough new youth to create a cohort. A
review of the Juvenile Justice Information System confirmed this claim.
Two cohorts were conducted during the review period. The first began 4/11/18 and ran until
8/7/18. This group began with ten (10) youth and ended with nine youth; one youth successfully
completed the program before the group ended. The second cohort began with ten (10) youth on
4/1/19. Each of those youths are still active in the group.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct internal fidelity monitoring. No fidelity monitoring
was conducted for Impact of Crime (IOC) during the review period. One other staff had been
trained in the curriculum but had never conducted groups, which the Clinical Director believed
was a requirement to conduct fidelity monitoring. A plan was put in place to begin fidelity
monitoring with trained staff and send additional staff to IOC training to avoid any further lapses in
fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions resulting from internal fidelity
monitoring. The Clinical Director conducts weekly supervision meetings with all clinical staff. If an
issue were noted on the internal fidelity monitoring forms, it is discussed during these meetings
and followed up on during the next internal fidelity monitoring. No fidelity monitoring was
completed during the review period; therefore, no points may be awarded in this section.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitator’s performance evaluation during the review period confirmed evaluation
of the staff’s performance in delivering Impact of Crime. The evaluation included a goal for
improvement during the next review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 20 youth in the
sample, 95% (19 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 20 youth in the
sample, 100% (20 of 20) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 4 youth
Moderate-High = 6 youth
High = 10 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (20 of 20) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (10 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment.
38 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 20 total youth sampled, all
20 included dosage with end dates in the EBS Module. Of those youth, 19 received at least the
recommended weeks of service. Youth in the sample completed between 12 and 22 weeks of
service, with an average of 19 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, all 20 included dosage in the EBS Module. Of those youth, 20 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 24 and 37 hours of
service, with an average of 33 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator, and by ensuring any
corrective action needed is identified, applied, and demonstrated.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
Highway 65 South
Sumatra, Florida 32335
Primary Service: Thinking for a Change
SPEP Review Date(s): May 14-17, 2019
Florida Department of Juvenile Justice
Report Date(s): May 17, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation provided confirmed the program has a training protocol for the primary service
Thinking for a Change, and three clinical staff are trained in the curriculum. The Clinical Director
was trained as a trainer in the curriculum in 2015, and two Clinical Coordinators were trained as
facilitators in 2017. Of the three staff, one Clinical Coordinator facilitated groups during the review
period, with the Clinical Director providing fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Thinking for a Change curriculum (T4C), version 4.0. The manual provides
a script for each of the 25 lessons. The youth are introduced to the skill and skill steps for each
lesson. There are facilitator notes located in the margins of each page. All skill cards, slides, and
homework sheets are available electronically, and each facilitator has access to the materials
required to conduct T4C. The manual and supplemental materials are detailed.
Documentation for the primary service is housed in a binder that is organized and easy to locate.
Cohorts stapled separately for easy review. Youth are placed in the intervention based on
identified criminogenic needs based on the Residential Positive Achievement Change Tool.
Youth not placed in Thinking for a Change are placed in Impact of Crime. Groups are held
Tuesdays and Thursdays, 9:45AM until 10:45AM.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
An observation of the final lesson of the cohort was observed during the annual review. The
facilitator began group with a welcome message and reviewed the group norms and expectations.
The group discussed the previous lesson’s homework, then the facilitator handed out certificates
and the youth celebrated their completion. The facilitator used positive affirmations and was
encouraging to all youth, taking care to note their successes through the course of the curriculum.
The facilitator was knowledgeable of the material and followed the lesson plan without reading
verbatim. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has a process in place to avoid gaps in service due to facilitator turnover. There
was no turnover during the review period. There was a gap in service from 9/15/18 through
1/2/19; however, this gap was due to there not being enough new youth to create a cohort. A
review of the Juvenile Justice Information System confirmed this claim.
Two cohorts were conducted during the review period. The first cohort began 4/7/18 with ten (10)
youth and ended 9/14/18 with nine youth; one youth successfully completed the program during
this service. The second cohort began 1/3/19 and ran until 5/16/19, with nine youth participating
for the duration.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct internal fidelity monitoring. The Clinical Director
conducts monthly fidelity monitoring for Thinking for a Change. Fidelity monitoring was conducted
each month, except for March 2019.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions resulting from internal fidelity
monitoring. The Clinical Director conducts weekly supervision meetings with all clinical staff. If an
issue were noted on the internal fidelity monitoring forms, it is discussed during these meetings
and followed up on during the next internal fidelity monitoring. A review of internal fidelity
monitoring forms during the review period concluded to corrective actions were noted nor taken.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitator’s performance evaluation during the review period confirmed evaluation
of the staff’s performance in delivering Thinking for a Change. The evaluation included a goal for
improvement during the next review period, comments, strengths, and an opportunity for the
facilitator to provide feedback.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 15 weeks for this type of service. Of the 14 youth in the
sample, 100% (14 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 45 contact hours for this type of service. Of the 14 youth in the
sample, 100% (14 of 14) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 8 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 57%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (14 of 14) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 57% (8 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 100 100 100%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Juvenile Unit for
Specialized Treatment.
38 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 14 total youth sampled, all
14 included dosage with end dates in the EBS Module. Of those youth, 14 received at least the
recommended weeks of service. Youth in the sample completed between 22 and 25 weeks of
service, with an average of 23 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 14 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 48 and 54 hours of
service, with an average of 50 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Date(s): 05/04/2020
Florida Department of Juvenile Justice
Report Date(s): 05/04/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There are seven trained staff for the Impact of Crime (IOC) curriculum. Two of the seven will not
be conducting groups on a regular basis. One trained employee left during the review period. One
staff member is the internal fidelity monitor for IOC. All staff have been formally trained. Four of
the seven trained facilitators have facilitated groups during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitators utilize the Impact of Crime (IOC), Addressing the Harm curriculum,
version 2.0.The program has a manual for the delivery of services and a workbook for the
students to use. The curriculum provides a script and instructions for seven chapters which
requires a minimum of thirty-two contact hours to facilitate. The curriculum includes
implementation guidelines, identifies the order the lessons are to be delivered, and provides
detailed instructions for the delivery of each lesson. Group is held on Tuesdays and Thursdays
from 3:00pm - 4:00pm. All youth are eligible to attend IOC groups. The IOC groups are being held
on a regular basis with fidelity.
The program has multiple opportunities for youth to participate in victim impact awareness. The
program maintains an aquaponics farm. The youth attend to and help plant all of the food that is
grown. They also feed the fish and assist in maintaining the equipment under the guidance of a
staff member. The food that is grown is provided to a local farmer's market. The money earned
from the food goes back into the aquaponics farm.
The program also has a wood/carpentry workshop for the youth. The youth builds and sells or
donates items made within the local community and to those in need. The youth assisted in
rebuilding many of the buildings on the campus after Hurricane Michael. The youth are currently
building offices within the main building on campus to include new classrooms, group rooms,
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
nurses station, and offices. This building project was observed during this visit to the program.
There is a welding workshop where the youth are learning the trade of welding. They make many
projects and sell them or donate them to local communities.
The youth also participate in the Boys and Girls club at the program. There are volunteer
opportunities provided throughout the year with this program where the youth are able to earn
community service hours. In the cohorts reviewed, there were guest speakers who come to the
program to speak with the youth on a regular basis. This was documented on the attendance
sheets for the group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed October 3, 2020 from 3:00pm to 4:00pm. There were
ten youth attending this group. All three facilitators were able to conduct the group. They did a
great job splitting up the lesson to ensure equal time facilitating. The group was working on the
first objective of chapter 2, personal accountability. The youth participated in the role play and
were provided feedback. The facilitators were well prepared and had all materials ready. The
facilitators did a good job of ensuring all the youth had an opportunity to participate. They moved
throughout the room to ensure youth were paying attention. The group ended the lesson by doing
a review of what was learned. The facilitators appeared to feel comfortable providing the lesson
and interacting with the youth. There were no concerns noted during the lesson.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol is for another trained staff member to continue facilitating the group in the
event of any turnover. One Impact of Crime (IOC) facilitator left during this review period; however
it did not affect the group. The gap in service during this review period was not due to facilitator
turnover, but due to a low census of youth coming into the program.
The program has conducted two full cohorts and a third cohort is due to end May 26th. The first
cohort was held April 1, 2019 through August 23, 2019. Ten youth started this group and ended
this group. The second cohort was held from September 29, 2019 through December 13, 2019.
Ten youth started the group. One youth left November 17, 2019 and did not successfully complete
the curriculum. The third cohort was started February 4, 2020 and is scheduled to end around
May 26, 2020. There are eight youth in the group.
Attendance sheets were provided for review. The attendance sheets were located in the program
SPEP binder. They were in order and able to be reviewed. The May attendance sheets were
missing during this review. The facilitator of the group reported they were moved into a storage
building and will need to be located. There was a gap in service between December 13, 2019
through February 4, 2020 due to a low census. All youth have been through IOC who were able.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Another trained facilitator conducts the internal fidelity monitoring for the Impact of Crime (IOC)
groups. Of the three trained staff who facilitated IOC, all months were accounted for during this
review period and provided for review.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a protocol for internal fidelity monitoring. This protocol was provided for review.
The therapist who conducts internal fidelity monitoring addresses any issues that arise during
group with the facilitator. It is documented on the internal fidelity monitoring form. The facilitator is
observed again and this information is documented. If the issue continues further action is taken.
Subsequent observations are also documented.
Upon review of the internal fidelity monitoring forms there were no corrective actions noted nor
taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Two of the four facilitators had an annual performance evaluation conducted during the review
period. One facilitator left during the review period and did not receive an annual evaluation. The
other facilitator has not been at the program long enough to have received an annual evaluation.
Staff are evaluated on their skill in delivering Impact of Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 17 youth in the
sample, 76% (13 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 6 youth
High = 9 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (16 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 53% (9 of 17) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 79 85 93%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment.
32 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 17 total youth sampled, all
17 included dosage with end dates in the EBS Module. Of those youth, 13 received at least the
recommended weeks of service. Youth in the sample completed between 11 and 21 weeks of
service, with an average of 18 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth, 17 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 30 and 36 hours of
service, with an average of 34 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Thinking for a Change
SPEP Review Date(s): 5/4/2020
Florida Department of Juvenile Justice
Report Date(s): 5/4/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There are three staff currently trained to facilitate the Thinking for a Change (T4C) curriculum. The
clinical director and two clinical staff are trained. One clinical coordinator is the main facilitator
who conducts T4C groups at the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Thinking for a Change curriculum (T4C), version 4.0. The manual provides
a script for each of the 25 lessons. The youth are introduced to the skill and skill steps for each
lesson. There are facilitator notes located in the margins of each page. All skill cards, slides, and
homework sheets are available electronically, and each facilitator has access to the materials
required to conduct T4C. The manual and supplemental materials are detailed.
Documentation for the primary service is housed in a binder that is organized and easy to locate.
Cohorts are stapled separately for easy review. Youth are placed in the intervention based on
identified criminogenic needs based on the Residential Assessment for Youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This group has not been observed during this review period. The group was not being held during
the quarterly review in October 2019. Due to COVID-19 protocols this reviewer was unable to
observe this group during the annual review. This indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The clinical director and two other clinical coordinators are trained in Thinking for a Change (T4C)
curriculum. Should there be turnover, the clinical director or clinical coordinator would take over
the group and complete it. There were no gaps in service due to facilitator turnover.
There has been one full cohort during this review period. A second cohort has been started. The
first cohort was held from May 21, 2019 through September 24, 2019. There were eight youth
who started the group and seven who successfully completed the group. One youth was
transferred to another program during the cohort. All attendance sheets were reviewed for this
cohort and the groups appeared to have been held with fidelity of the model.
The second cohort started January 27, 2020 and is still currently meeting. One group was held for
40 minutes on February 21, 2020. The requirement for this group is to be held for one hour, at a
minimum. This was the only attendance form showing a group time of less than one hour. It is
difficult to tell what times of the day the groups are held. There was no consistency in times
throughout the attendance sheets reviewed. The days groups are held, based on attendance
sheets, were also difficult to discern. Some weeks groups were Wednesdays and Fridays and
some weeks groups were held Tuesdays and Thursdays.
Pursuant to a discussion with the clinical director, there was a gap in service between September
2019 and January 2020 due to low census and minimal youth entering the program. All youth
have been able to attend T4C. This was verified through a review of the attendance sheets and
information entered into the Evidence-Based Services Module in the Juvenile Justice Information
System.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a policy in place for internal fideilty monitoring and it was reviewed for this
annual review. The clinical director conducts internal fidelity monitoring for the Thinking for a
Change (T4C) groups. All internal fidelity monitoring forms were reviewed and accounted for
during this review period, with no exceptions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the clinical director to address any issues during group with the
facilitator. There is a section on the internal fidelity monitoring form to document any issues and
corrective actions. The facilitator will be observed again and documented on the fidelity form. If
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
the issue continues further action is taken and documented. There were no corrective actions for
this facilitator during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations include a section for group facilitation. The evaluation
mentions the specific curriculua the staff member facilitates. The facilitator for Thinking for a
Change group received their annual performance evaluation during this review period. It includes
a review of group facilitation for the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 15 weeks for this type of service. Of the 15 youth in the
sample, 60% (9 of 15) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 5 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 5 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (14 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 33% (5 of 15) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 20 25 80%
Totals 81 100 81%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Juvenile Unit for
Specialized Treatment.
32 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 15 total youth sampled, all
15 included dosage with end dates in the EBS Module. Of those youth, 9 received at least the
recommended weeks of service. Youth in the sample completed between 11 and 20 weeks of
service, with an average of 17 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, all 15 included dosage in the EBS Module. Of those youth, 0 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 36 and 42 hours of
service, with an average of 40 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9