summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunity Investments
(Contract Provider)
8301 South Palm Drive,
Pembroke Pines , Florida 33025
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): April 11‐14, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/14/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Broward Youth Treatment Center MQI Program Code: 1269
Provider Name: Youth Opportunity Investments Contract Number: 10440
Location: Broward County / Circuit: 17 Number of Beds: 28
Review Date(s): April 11-14, 2017 Lead Reviewer Code: 50
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program had a training protocol and there were three staff
who have received formal training from a qualified trainer to become a group facilitator in the
primary service Impact of Crime (IOC). One case manager was trained on October 27, 2006, one
youth care worker was trained on December 4, 2015 and one clinical director was trained on
August 14, 2015. The youth care worker was facilitating group during the review period until
December 20,2016. The youth care worker left the program and the case manager took over
facilitating the group.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilized the Impact of Crime (IOC): Addressing the Harm curriculum. The
program has a manual for the delivery of services and a workbook for the students to use. A copy
of the manual and workbook was provided for the team to reference during the review. The
curriculum provides a script and instructions for seven chapters. The curriculum includes
implementation guidelines, identifies the order the lessons are to be delivered, and provides
detailed instructions for the delivery of each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The team observed an Impact of Crime (IOC) session during the annual compliance review. The
group was held in the breezeway because the weather was nice and the youth enjoyed having
group outdoors. The facilitator was prepared for the lesson, reviewed the objectives, words to
know and information from the previous class. The facilitator demonstrated knowledge of the
material and was able to keep the participants interested. All youth were on task, focused and
involved in the lesson. The facilitator organized a fun competition involving the exercise in the
workbook and gave candy to the youth as a reward for participation. Affirmations were given to
the youth throughout the session and there were no behavior management issues.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation confirmed there was approximately a two-week gap in service
delivery which adversely affected youth due to turnover when one of facilitators left the program
during a cohort of Impact of Crime. In the event of facilitator turnover, and to minimize gaps in
service, the program has made arrangements to have five staff to be trained in Impact of Crime in
May 2017.
The program can earn 2 points by minimizing facilitator turnover and gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
During the review period, documentation indicated the clinical director who performed fidelity
monitoring was trained by a qualified trainer to teach the primary service. The clinical director
conducted a fidelity monitoring every month and the last one was conducted on December 1,
2016. The clinical director resigned from the program on December 2017. The current clinical
director has not been trained in Impact of Crime (IOC) and is scheduled to attend training class in
May 2017. Internal fidelity checklists are utilized to capture specifics of the primary service. On the
checklist, there is a corrective action follow-up component which incorporated any applicable
recommendations identified during fidelity monitoring. These documents were maintained in a
IOC binder along with the IOC sign-in sheets.
The program can earn 2 points by ensuring internal fidelity is conducted at least monthly for all
facilitators by an individual trained to deliver IOC.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Reviewed documentation during the review period and verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. The
program has not had any applicable issues requiring corrective actions. The completed fidelity
monitoring forms recorded the observer’s feedback and recommendations, and ideas for
improvement in the delivery of the primary service Impact of Crime (IOC).
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Youth Opportunity Investments program took over as the provider for Broward Youth Treatment
Center on August 25, 2016 and an annual employee performance evaluation had not yet been
conducted at the time of the review.
The program can earn 1 point by ensuring all facilitators' performance evaluations contains an
assessment of their delivery of Impact of Crime (IOC).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 13 youth in the
sample, 38% (5 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 13 youth in the
sample, 38% (5 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 5 youth
High = 7 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 54%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 54% (7 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 25 25 100%
Totals 59 85 69%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Broward Youth
Treatment Center.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 13 total youth sampled, only
5 included dosage with end dates in the EBS Module. Of those youth with esnd dates, all 5
received at least the recommended weeks of service. Youth in the sample completed exactly 17
weeks of service.
The program earned 2 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, 5 received at least the recommended hours of service. Youth in the sample completed
between 5 and 31 hours of service, with an average of 18 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Broward Youth Treatment Center can optimize their SPEP Quality of Service Delivery score by
minimizing facilitator turnover and gaps in service, by ensuring internal fidelity is conducted at
least monthly for all facilitators by an individual trained to deliver IOC, and by ensuring all
facilitators' performance evaluations contains an assessment of their delivery of Impact of Crime
(IOC).
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
data is entered accurately in the EBS Module and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

report_type: SPEP

program_name_raw: Broward Youth Treatment Center - YOI

program_status_note: Closed

service_type: Impact of Crime


state: FL

facility_name: Broward Youth Treatment Center

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunity Investments
(Contract Provider)
8301 South Palm Drive,
Pembroke Pines , Florida 33025
Primary Service: Thinking For A Change (T4C)
SPEP Review Date(s): April 11-14, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/14/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Broward Youth Treatment Center MQI Program Code: 1269
Provider Name: Youth Opportunity Investments Contract Number: 10440
Location: Broward County / Circuit: 17 Number of Beds: 28
Review Date(s): April 11-14, 2017 Lead Reviewer Code: 50
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking For A Change (T4C). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program had a training protocol and there were two
staff who have received formal training from a qualified trainer to become group facilitators in
Thinking for a Change (T4C). One youth care worker was trained on June 6, 2014, and the prior
clinical director was trained on May 30, 2013. The youth care worker facilitated groups during this
review period. The previous clinical director conducted fidelity monitoring every month until
December, 2016.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the curriculum of Thinking for a Change (T4C) 3.1 to qualifying youth admitted
to the program. T4C is comprised of 25 lessons that build upon each other, and contains
appendices that can be used to craft an aftercare program to meet ongoing cognitive behavioral
needs of the group. Not all lessons can be completed in one session, so a typical delivery cycle
may take 30 sessions. Sessions should last between one and two hours. Ideally, the curriculum is
delivered two times per week, with a minimum recommended dosage of once per week and a
maximum of three times per week. Participants must be granted time to complete mandatory
homework between each lesson. T4C is intended for groups of eight to twelve and should be
delivered only by trained facilitators. Due to its integrated structure, T4C is a closed group,
meaning members need to start at the beginning of a cycle, and may not join the group mid-
stream. The curriculum provides a script and instructions for twenty-five lessons. The curriculum
includes implementation guidelines, identifies the order the lessons are to be delivered, and
contains detailed instructions for each lesson.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of a T4C group was made on April 13, 2017. There were eight youth in
attendance. The facilitator seemed comfortable presenting in front of the group and was familiar
with the material. The room in which the group was held was conducive to learning. One youth
assisted the facilitator with the presentation and the role play demonstrations. All the youth
enjoyed the role play activity and were having fun. At one-point youth went off topic, but the
facilitator was able to bring the youth back on track and focus on the lesson. The facilitator used
affirmations and there were no behavior management issues during group. The facilitator did not
have all of the presentation material to effectively conduct group. Chart paper, makers and a white
board was available, but a laptop and projector was not available to display the PowerPoint slides.
The facilitator lost access to the National Institute of Corrections (NIC) website when Thinking for
a Change was updated from 3.1 to 4.0. A request was submitted to allow the facilitator to have
access to the website.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and confirmed there have been no gaps in service delivery and no
turnover of facilitators which adversely affected T4C groups during this review period. In the event
of facilitator turnover, the program is in the process to have five additional staff to be trained in the
Thinking for a Change curriculum.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed internal fidelity monitoring reports for this review period. Documentation confirmed the
staff member who performed fidelity monitoring was trained by a qualified trainer to teach the
primary service. The program internal fidelity monitoring was conducted by the previous clinical
director for the one facilitator once a month with the last valid fidelity monitoring completed on
December 14, 2016. The new clinical director conducted fidelity monitoring for the months of
February and March 2017, but is not certified in the Thinking for a Change curriculum. The
program is in the process to have five additional staff to be trained as Thinking for a Change
facilitators. The internal fidelity checklists are utilized to capture specifics of the primary service.
On the checklist, there is a corrective action follow-up component which incorporated any
applicable recommendations identified during fidelity monitoring. These documents were
maintained in a Thinking for a Change binder along with the T4C sign-in sheets.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 2 points by ensuring internal fidelity is conducted at least monthly for all
facilitators by an individual trained to deliver T4C.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The team reviewed documentation for the last twelve months and verified the program has a
process for the application of corrective actions based on the results of the fidelity monitoring
conducted. The program has not had any applicable issues requiring corrective actions. The
completed fidelity monitoring forms recorded the observer’s feedback and recommendations, and
ideas for improvement in the delivery of the primary service Thinking for a Chance (T4C).
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Youth Opportunity Investments program took over as the provider for Broward Youth Treatment
Center on August 25, 2016 and an annual employee performance evaluation had not yet been
conducted at the time of the review.
The program can earn 1 point by ensuring all facilitators' performance evaluations contains an
assessment of their delivery of Thinking For A Change (T4C).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 15 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 60%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 2 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 22%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 2 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (8 of 9) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 22% (2 of 9) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 15 25 60%
Totals 70 100 70%
This SPEP report evaluates Thinking For A Change (T4C), an intervention delivered at Broward
Youth Treatment Center.
The program scored High for Quality of Service Delivery.
The program earned no points for Amount of Service: Duration. Of the 9 total youth sampled,
none received at least the recommended weeks of service. Sampled youth completed 14.7 weeks
of service - just shy of 15 weeks of service.
The program earned no points for Amount of Service: Contact Hours. Of the 9 total youth
sampled, none received at least the recommended hours of service. All youth in the sample
completed between 22 and 24 hours of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 15 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Broward Youth Treatment Center can optimize their SPEP Quality of Service Delivery score by
ensuring internal fidelity is conducted at least monthly for all facilitators by an individual trained to
deliver T4C and by ensuring all facilitators' performance evaluations contains an assessment of
their delivery of Thinking For A Change (T4C).
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
data is entered accurately in the EBS Module and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

service_type: Thinking for a Change


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunites Investments, LLC.
(Contract Provider)
8301 South Palm Drive, Building 2
Pembroke Pines, Florida 33025
Primary Service: Impact of Crime
SPEP Review Date(s): January 30 - February 2, 2018
Florida Department of Juvenile Justice
Report Date(s): February 5, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Broward Youth Treatment Center MQI Program Code: 1269
Provider Name: Youth Opportunities Investments, LLC. Contract Number: 10440
Location: Broward County / Circuit: 17 Number of Beds: 40
Review Date(s): January 30 - February 2, 2018 Lead Reviewer Code: 142
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Impact of Crime is based on the belief of crime is more than just a legal definition, but rather crime
affects the victim, the offender, their families, and the community. The harm caused brings with it
the moral responsibility to all involved. The main focus of restorative justice is to help offenders
understand the harm they have created and then assisting them in taking personal accountability
for their actions. Together, the victim, offender, and the community arrive at a viable solution to
the repair the harm caused and the offender, once the harm is repaired, is subsequently
reintegrated back into the community. Facilitators are encouraged to bring in victim impact
speakers throughout the course of the curriculum. The Standardized Program Evaluation Protocol
(SPEP) review encompasses twelve months of SPEP records and examinations. The primary
service is being facilitated by one case manager who was trained by a qualified trainer in the
primary service. The program has an additional case manager trained in the primary service as
well. The clinical director was trained in the primary service and completes internal fidelity
monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes Impact of Crime: Addressing the Harm to Victims and the Community. The
curriculum was last updated in 2009. The manual is highly detailed providing a facilitators manual
and student workbook. The facilitators manual provides a script for the facilitator to follow,
important points to stress during the delivery of the lessons, individual lesson plans, and a long-
term plan for delivery of the primary service.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This technical assistance specialist did observe an Impact of Crime group during the week of the
on-site annual compliance review. The observer was provided a copy of the lesson plan and the
homework being assigned to the youth. The facilitator followed the script and kept the group
moving and on task. The facilitator was also able to incorporate audio visual presentations to keep
the interest of the group and the lesson moving along.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewd documentation coupled with an informal interview with the program's clinical director
confirmed there was no turnover of staff which directly affected the delivery of the primary service.
At the end of the last cohort, a previous facilitator transferred to deliver another service, but a new
facilitator was trained in the service during downtime and successfully began a new cohort in
December of 2017.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided documentation of internal fidelity monitoring which took place monthly.
The monitoring was performed by a trained facilitator and spaces were provided for observation
feedback, facilitator’s ideas for improvement, and corrective actions, if needed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process to create corrective action plans for facilitators who need skill
improvements. During the review period, none of the internal fidelity monitoring forms
documented a need for corrective actions.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of one of the facilitator's annual performance review confirmed the employee was rated
on their effectiveness as a facilitator. The facilitator currently delivering the primary service has
not been employed long enough for an annual performance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 12 youth in the
sample, 0% (0 of 12) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 12 youth in the
sample, 0% (0 of 12) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 9 youth
Total Youth in Sample = 12 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 75%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 12 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (12 of 12) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 75% (9 of 12) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Broward Youth
Treatment Center.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration and Amount of Service: Contact
Hours. There was no data entered into the Juvenile Justice Information System (JJIS) Evidence-
Based Services Module (EBS) Module for any youth. The program must enter dosage data into
JJIS in order to receive any score in this area.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
RECOMMENDATION(S):
Broward Youth Treatment Center can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunity Investments, LLC.
(Contract Provider)
8301 South Palm Drive, Bldg 2
Pembroke Pines, Florida 33025
Primary Service: Thinking for a Change
SPEP Review Date(s): January 30 - February 2, 2018
Florida Department of Juvenile Justice
Report Date(s): February 5, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Broward Youth Treatment Center MQI Program Code: 1269
Provider Name: Youth Opportunity Investments, LLC. Contract Number: 10440
Location: Broward County / Circuit: 17 Number of Beds: 40
Review Date(s): January 30 - February 2, 2018 Lead Reviewer Code: 142
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve months of
SPEP records. The program is currently utilizing the Thinking for a Change 4.0 curriculum. A new
cohort began in in late December of 2017 and is ongoing. The program completed three other
cohorts during the previous twelve months. Groups were conducted as closed groups, meaning
once a group is commenced, no new youth can be added to the group. Youth are selected for the
group based on a review of the youth's needs at the time of their acceptance into the program.
This review team consists of the clinical director and case managers. The program provided
documentation of a case manager and a direct care staff worker being trained in the primary
service and are currently co-facilitating the group. Additionally, the clinical director was trained in
the primary service and is providing internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provides the evidence based curriculum Thinking for a Change 4.0 to qualifying
youth admitted to the program. The program utilizes a highly detailed commercially produced
manual which provides lesson plans, facilitator notes, ideas to facilitate the curriculum, and long-
term planning schedules.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The technical assistant specialist observed a Thinking for a Change group while on-site during the
annual compliance review. The facilitators followed the script closely, but occasionally added their
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
own words to give meaning and context to the curriculum. The facilitators utilized current
technology to present the PowerPoint slides utilized by the curriculum.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The technical assistance specialist reviewed documentation and interviewed the clinical director.
During the past twelve months, there has been no facilitator turnover which affected the delivery
of the primary service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program received on-site training in the primary service in June of 2017. Due to various
technical issues, computer access to the curriculum was delayed until September of 2017.
Internal fidelity monitoring took place in December of 2017 and January of 2018.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The technical assistance specialist conducted a review of the two months of fidelity monitoring
which took place at the program. The review verified the program has a process for the
application of corrective actions based upon the results of fidelity monitoring conducted. The
program documented no issues requiring corrective action on the internal fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The technical assistant specialist reviewed the annual reviews of staff who were facilitating the
primary service during the review period. The performance evaluations had sections noting the
employee was a facilitator of the primary service and rated their skills in delivering the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 9 youth in the
sample, 0% (0 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 2 youth
High = 6 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 67%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (8 of 9) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 67% (6 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 78 100 78%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Broward Youth
Treatment Center.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration and Amount of Service: Contact
Hours. There was no data entered into the Juvenile Justice Information System (JJIS) Evidence-
Based Services Module (EBS) Module for 6 of the 9 youth sampled. The program must enter
dosage data into JJIS in order to receive any score in this area.
The program earned 0 points for Amount of Service: Duration. Of the 9 total youth sampled, none
included dosage with end dates in the EBS Module.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 9 total youth sampled,
only 3 included dosage in the EBS Module. Of those youth with correct dosage, 0 received at
least the recommended hours of service. Youth in the sample completed 26 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Broward Youth Treatment Center can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunity Investments, LLC.
(Contract Provider)
8301 South Palm Drive, Bldg. 2
Pembroke Pines, Florida 33025
Primary Service: Impact of Crime
SPEP Review Date(s): October 23-26, 2018
Florida Department of Juvenile Justice
Report Date(s): October 29, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Impact of Crime is based on the belief that crime is more than just a legal definition, but rather
crime affects the victim, the offender, their families, and the community. The harm caused brings
with it the moral responsibility to all involved. The focus of Impact of Crime is to help offenders
understand the harm they have created, and then assisting them in taking personal accountability
for their actions. Together, the victim, offender, and the community arrive at a viable solution to
the repair the harm caused and the offender. Once the harm is repaired, the offender
subsequently reintegrates back into the community. Facilitators are encouraged to bring in victim
impact speakers throughout the course of the curriculum. The curriculum also allows for the
planning of a community service project as a capstone to the primary service. Youth are able to
provide community service in the community, or on site at the program to help to heal some of the
harm done by delinquent acts.
The SPEP review encompasses twelve (12) months of records examination. According to training
records maintained in the Department's learning management system (SkillPro), all facilitators,
and all staff providing internal fidelity monitoring were trained in the primary service by a certified
master trainer. Those employees included: supervisor of case management, and three case
managers.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the commercially produced curriculum Impact of Crime: Addressing the
Harm to Victims and the Community 2.0. The curriculum was updated in 2011. It is a highly
detailed curriculum providing a facilitator's manual and student workbook. The facilitator's manual
provides lessons plans, classroom management ideas, and tips for engaging youth discussions.
The student manual provides reproducible worksheets to help facilitate group discussions, and
review concepts learned during the group sessions. Homework materials are also utilized in the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
student manual to reinforce learned concepts. Impact of Crime is a closed group meaning once a
group commences, new youth cannot be added to the group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
During the annual review, the Technical Assistance Specialist observed the primary service. The
group consisted of ten (10) youth and was conducted on the covered patio outside the rear of the
building. The cooler weather allowed for group to be held outside, which the youth stated they
enjoyed. Although the group was outside, the youth stayed on task, and were engaged in the
lesson. They were not distracted by being outside, nor by new stimuli. The group was co-
facilitated by a newly trained case manager and a more experience case manager. An outline of
the lesson was given to the review team. The facilitators read the script, but also stopped and
discussed what was read with the youth to facilitate large group discussion, and check for youth
understanding. The facilitators explained the contents of the primary service in their own words.
The lesson was conducted within the required timeframes, and no behavioral issues were
observed during the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
In reviewing documentation, and an informal interview with clinical director and case management
supervisor, there was no facilitator turnover during the review period that affected delivery of the
primary service. In the event of a facilitator leaving the program, another facilitator trained in the
primary service would assume facilitator duties.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program completed one cohort of the primary service during the review period. Two new
cohorts had begun two weeks prior to the annual review and was ongoing. The curriculum calls
for a minimum of thirty-two (32) sessions in sixteen (16) weeks to complete one cohort. According
to documentation provided by the program, no internal fidelity monitoring took place during the
last year.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has an implementation procedure of correction actions for facilitator if needed. In
informal discussion with the clinical director and case management supervisor, in case of need of
corrective action for facilitators, a conference is held with the supervisor and facilitator. A
corrective action plan is developed cooperatively between the parties, and additional internal
fidelity monitoring takes place to ensure the improvement of the facilitator's skill set.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations were reviewed, and it was documented the facilitators were
providing groups in the primary service. The evaluations also rated the skill of the facilitators in
service delivery.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 55 youth in the
sample, 0% (0 of 55) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 55 youth in the
sample, 16% (9 of 55) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 98%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 16 youth
High = 36 youth
Total Youth in Sample = 55 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 65%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 36 youth
Total Youth in Sample = 55 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 98% (54 of 55) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 65% (36 of 55) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 55 85 65%
This SPEP report evaluates Impact of Crime, an intervention delivered at Broward Youth
Treatment Center.
59 youth completed the program during the review period. Of those youth, 55 had not previously
received this intervention in another setting.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 55 total youth sampled, only
17 received this intervention. Of those youth, 0 received at least the recommended weeks of
service. Youth in the sample completed between 6 and 11 weeks of service, with an average of 9
weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 55 total youth
sampled, only 17 received this intervention. Of those youth, 9 received at least the recommended
hours of service. Youth in the sample completed between 4 and 36 hours of service, with an
average of 19 hours.
Due to a technical issue with the Evidence-Based Services module, data entered for each cohort
of each primary service was removed and unavailable for evaluation. The program became
aware of the issue and has taken measures to prevent this error in the future. A manual review of
sign in sheets was conducted to calculate these totals.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Broward Youth Treatment Center can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator, and by ensuring any
corrective action needed is identified, applied, and demonstrated.
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Broward Youth Treatment Center
Youth Opportunity Investment, LLC
(Contract Provider)
8301 South Palm Drive, Bldg. 2
Pembroke Pines, Florida 33025
Primary Service: Thinking for a Change
SPEP Review Date(s): October 23-26, 2018
Florida Department of Juvenile Justice
Report Date(s): October 29, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve months of
SPEP records. The program is currently utilizing the Thinking for a Change (T4C) 4.0 curriculum.
T4C is an evidenced-based practice. T4C utilizes a problem-solving program with both cognitive
restructuring, and social skills interventions bringing the identification of thinking, beliefs, attitudes
and values to behavior. Effort is focused on helping youth become aware of their thoughts,
feelings and beliefs in particular circumstances. All staff either facilitating groups or providing
internal fidelity monitoring were trained in the primary service by a master trainer. The trained
staff members include: two case managers, two youth care workers, one youth care worker
supervisor, and the clinical director.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program was able to provide the manuals for Thinking for a Change (T4C) 4.0. The program
utilizes a highly detailed, commercially produced manual that provides lesson plans, facilitator
notes, ideas to facilitate the curriculum, and long-term planning schedules. Groups are conducted
as closed groups, meaning once a group commenced, no new youth can be added to the group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Staff was able to observe a group during the annual review. The group
consisted of ten youth in a classroom setting. The group was co-faciliated by an experienced
case manager and a newly trained youth care worker, who was facilitating her first cohort. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
facilitators read the script to the youth, but also was able to stop from the script and add clarity
and check for understanding by the youth. No behavior issues were observed during the group.
The facilitators role-played the new skill, and each youth was given the opportunity in pairs to
practice the skill. Homework was also assigned at the conclusion of the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to documentation provided by the program, and informal interview with the case
manager supervisor, no facilitatior turnover occurred which affected the delivery of the primary
service. Documentation shows a cohort was completed from April 2018 until July 2018. Two new
cohorts just began two weeks before the annual review.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
According to documentation provided by the program, and informal interview with case manager
supervisor, no internal fidelity montioring took place during the reviewed time periods. According
to an informal interview with the clinical director, it was learned she had just received training in
the primary service. She was recently hired, and was not able to conduct internal monitoring until
just one week before annual review. Clinical director has a plan for monthly internal fidelity
monitorings moving forward.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Although no internal fidelity monitoring took place during the review period, an informal interview
with the clinical director confirmed the program has a policy in place to address facilitators in need
of corretive actions. In the event a facilitator was in need of skill enhancement, a conference
would be held with the facilitator and supervisor to develop a plan in which the facilitator's skills
would be improved. Additional internal fidelity monitoring would then take place to ensure
compliance.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of annual performance evaluations for the staff providing the primary service revealed
the evaluations documented the staff as facilitators of the primary service. Additionally, the
evaluations rated the staff in their skill as facilitators in the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 29 youth in the
sample, 24% (7 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 45 contact hours for this type of service. Of the 29 youth in the
sample, 24% (7 of 29) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 97%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 11 youth
High = 17 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 59%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 17 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 97% (28 of 29) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 59% (17 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 25 25 100%
Totals 74 100 74%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Broward Youth
Treatment Center.
59 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 29 total youth sampled, 7
received at least the recommended weeks of service. Youth in the sample completed between 5
and 15 weeks of service, with an average of 11 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, 7 received at least the recommended hours of service. Youth in the sample completed
between 12 and 57 hours of service, with an average of 38 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Due to a technical issue with the Evidence-Based Services module, data entered for each cohort
of each primary service was removed and unavailable for evaluation. The program became
aware of the issue and has taken measures to prevent this error in the future. A manual review of
sign in sheets was conducted to calculate these totals.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Broward Youth Treatment Center can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator, and by ensuring any
corrective action needed is identified, applied, and demonstrated.
Broward Youth Treatment Center can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9