state: FL

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel TSI of Florida
(Contract Provider)
9680 Fairground Road,
Palm Beach, Florida 33411
Primary Service: Aggression Replacement Training (ART)
SPEP Review Date(s): April 18, 2017 - April 21, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/9/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1417
Provider Name: Sequel TSI of Florida Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 70
Review Date(s): April 18, 2017 - April 21, 2017 Lead Reviewer Code: 139
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training (ART). The program
was awarded 30 points because the Primary Service is identified as a Group 5 Service. The
specific Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary
Service was identified as this type of service as it is intended to correct faulty cognitions and
perceptions and provides skills individuals can use to monitor thought patterns and correct
behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program had a training protocol and there were eight
staff for which have received formal training from a qualified trainer to become group facilitators in
Aggression Replacement Training (ART). One clinical director was trained on March 6, 2015 and
two therapists were trained on December 17, 2014 and one therapist was trained on August 28,
2015 and three therapists was trained on May 19, 2015. Five therapists facilitated groups during
the last twelve months. The clinical director provided fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the evidenced-based curriculum Aggression Replacement Training (ART).
ART consists of ten weeks with thirty sessions of intervention training, and is divided into three
components—social skills training, anger-control training, and training in moral reasoning. Clients
attend a one-hour session in each of these components three times a week. Incremental learning,
reinforcement techniques, and guided group discussions enhance skill acquisition and reinforce
the lessons in the curriculum.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Observed an Aggression Replacement Training (ART) session during the annual compliance
review. The facilitator demonstrated knowledge of the content and was comfortable facilitating a
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
group of youth. The problem situation was presented to the group and was made relevant to the
group members lives. The facilitator challenged thinking errors in a respectful way. Individuals
were praised for positive answers. All group members were involved in the discussion and there
were no behavior issues during group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and confirmed there have been no gaps in service delivery and no
turnover of facilitators which adversely affected ART groups during the last twelve months. In the
event of facilitator turnover, the co-facilitator or another trained facilitator will assume the role of
the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed twelve months of internal fidelity monitoring reports. Documentation confirmed staff
members who performed fidelity monitoring were trained by a qualified trainer to teach the primary
service. The program internal fidelity monitoring was conducted for all the facilitators once a
month within the last twelve months. Internal fidelity checklists are utilized to capture specifics of
the primary service. On the checklist, there is a corrective action follow-up component which
incorporated any applicable recommendations identified during fidelity monitoring. These
documents were maintained in a ART binder along with the sign-in sheets.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Reviewed documentation for the last twelve months, verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. The
program has not had any applicable issues requiring corrective actions. The completed fidelity
monitoring forms recorded the observer’s feedback and recommendations, and ideas for
improvement in the delivery of the primary service Aggression Replacement Training (ART).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of three performance evaluation documents confirmed the facilitators were
acknowledged on the annual performance evaluation in delivering the primary service Aggression
Replacement Training (ART). This included feedback and recommendations of the facilitator’s
proficiency in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 25 youth in the
sample, 96% (24 of 25) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 25 youth in the
sample, 100% (25 of 25) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 84%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 3 youth
High = 13 youth
Total Youth in Sample = 25 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 52%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 13 youth
Total Youth in Sample = 25 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 84% (21 of 25) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 52% (13 of 25) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 93 100 93%
This SPEP report evaluates Aggression Replacement Training (ART), an intervention delivered at
Palm Beach Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 25 total youth sampled, all
included dosage with end dates in the EBS Module. Of those youth with correct dosage, 24
received at least the recommended weeks of service. Youth in the sample completed between 7
and 10 weeks of service, with an average of 10 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 25 total youth
sampled, all included dosage in the EBS Module. Of those youth with correct dosage, all received
at least the recommended hours of service. Youth in the sample completed between 24 and 30
hours of service, with an average of 30 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Palm Beach Youth Academy - Sequel

program_status_note: Closed

service_type: Aggression Replacement Training


facility_name: Palm Beach Youth Academy

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel TSI of Florida
(Contract Provider)
9680 Fairground Road,
Palm Beach, Florida 33411
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): April 18, 2017 ‐ April 21, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/9/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1417
Provider Name: Sequel TSI of Florida Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 70
Review Date(s): April 18, 2017 - April 21, 2017 Lead Reviewer Code: 139
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee 1 # Clinical Staff 2 # Other (listed by title):
DMHCA or designee # Healthcare Staff Recreational Therapists
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 10 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program had a training protocol and there were six staff
who have received formal training from a qualified trainer to become a group facilitator in the
primary service Impact of Crime (IOC). Two recreational therapists were trained on July 22, 2016,
one case manager was trained on November 22, 2013, one youth care worker supervisor was
trained on November 20, 2014, one clinical director was trained on February 20, 2015 and one
director of case management was trained on December 15, 2011. The two recreational therapists
have facilitated group within the last twelve months. The clinical director and director of case
management conducted fidelity monitoring. The case manager has facilitated groups but is no
longer employed at the facility and the youth care worker supervisor has facilitated groups within
the last twelve months but no longer facilitates groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilized the Impact of Crime (IOC): Addressing the Harm curriculum. The
program has a manual for the delivery of services and a workbook for the students to use. A copy
of the manual and workbook was provided for the team to reference during the review. The
curriculum provides a script and instructions for seven chapters taking approximately twenty-four
sessions to facilitate. The new SPEP requires a minimum of thirty-two contact hours. The
curriculum includes implementation guidelines, identifies the order the lessons are to be delivered,
and provides detailed instructions for the delivery of each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Observed an Impact of Crime (IOC) session during the annual compliance review. The group was
held in the classroom and was conducive to learning. The facilitator was prepared for the lesson
and was comfortable presenting the lesson to the youth. The facilitator demonstrated knowledge
of the material and was able to keep the participants interested. All youth were on task, focused
and involved in the lesson. Affirmations were given to the youth throughout the session and there
were no behavior management issues. All the required visual aids were not available; the
facilitator had chart paper and makers but the program did not provide a laptop and projector to
display the Power Point slides and show the videos. The clinical director stated that the
equipment was stolen but have not been replaced at the time of the review.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and an interview with the clinical director confirmed there have been
no gaps in service delivery and no turnover of facilitators since the primary service Impact of
Crime (IOC) which adversely affected youth during the last twelve months. In the event of
facilitator turnover, another trained staff will assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
During the review period, documentation indicated the clinical director and the director of case
management who performed fidelity monitoring were trained by a qualified trainer to teach the
primary service. The clinical director and director of case management conducted a fidelity
monitoring every month for all of the facilitators. On the checklist, there is a corrective action
follow-up component which incorporated any applicable recommendations identified during fidelity
monitoring. These documents were maintained in a IOC binder along with the IOC sign-in sheets.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Reviewed documentation during the review period and verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. The
program has not had any applicable issues requiring corrective actions. The completed fidelity
monitoring forms recorded the observer’s feedback and recommendations, and ideas for
improvement in the delivery of the primary service Impact of Crime (IOC).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of two recreational therapists, and director of case management performance
evaluations confirmed the facilitators were acknowledged on their annual performance evaluation
in delivering the primary service Impact of Crime (IOC). This included feedback and
recommendations of the facilitators’ proficiency in delivering the primary service. The clinical
director was not acknowledged in the annual performance evaluation for conducting fidelity
monitoring on the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 13 youth in the
sample, 85% (11 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 13 youth in the
sample, 92% (12 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 77%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 3 youth
Moderate-High = 2 youth
High = 5 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 77% (10 of 13) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 38% (5 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 76 85 89%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Palm Beach
Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 13 total youth sampled, all
included dosage with end dates in the EBS Module. Of those youth with correct dosage, 11
received at least the recommended weeks of service. Youth in the sample completed between 12
and 20 weeks of service, with an average of 18 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all included dosage in the EBS Module. Of those youth with correct dosage, 12 received
at least the recommended hours of service. Youth in the sample completed between 20 and 43
hours of service, with an average of 39 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel TSI of Florida
(Contract Provider)
9680 Fairground Road,
Palm Beach, Florida 33411
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): April 18, 2017 ‐ April 21, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/9/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1417
Provider Name: Sequel TSI of Florida Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 70
Review Date(s): April 18, 2017 - April 21, 2017 Lead Reviewer Code: 139
Persons Interviewed
Program Director Corporate QI/QA staff 4 # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 10 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change (T4C). The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program had a training protocol and there were ten
staff for which have received formal training from a qualified trainer to become group facilitators in
Thinking for a Change (T4C). One case manager and transition coordinator were trained on
January 13, 2017, three case managers and one transition coordinator were trained on July 26,
2016. One case manager was trained on December 8, 2005. One case manager and the director
of case management were trained November 21, 2014. The clinical director was trained on March
27, 2015. Eight trained staff facilitated groups during the last twelve months. At the time of the
review there were two trained staff facilitating group and the clinical director and director of case
management were conducting fidelity monitoring reviews.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the evidenced-based curriculum of Thinking for a Change (T4C) 4.0 to
qualifying youth admitted to the program. T4C is comprised of 25 lessons that build upon each
other, and contains appendices that can be used to craft an aftercare program to meet ongoing
cognitive behavioral needs of the group. Not all lessons can be completed in one session, so a
typical delivery cycle may take thirty sessions. Sessions should last between one and two hours.
Ideally, the curriculum is delivered two times per week, with a minimum recommended dosage of
once per week and a maximum of three times per week. Participants must be granted time to
complete mandatory homework between each lesson.T4C is intended for groups of eight to
twelve and should be delivered only by trained facilitators. Due to its integrated structure, T4C is a
closed group, meaning members need to start at the beginning of a cycle, and may not join the
group mid-stream. The curriculum provides a script and instructions for twenty-five lessons. The
curriculum includes implementation guidelines, identifies the order the lessons are to be delivered,
and detailed instructions for each lesson.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation was made during the annual compliance review. The lesson preparation was
adequate. The facilitators printed out copies of the Power Point slides and gave copies to the
youth. They also displayed the printouts on the white board in front of the class but was difficult to
see from the back of the room. The required audio visual equipment was not available to properly
display the Power Point slides. The clinical director stated that the laptop and projector were
stolen and had not been replaced at the time of the review. The facilitators demonstrated
knowledge of the material and followed the curriculum. They modeled the new skill correctly and
received positive feedback from the youth. The group was held in the classroom which was
comfortable and conducive to learning. At the beginning of the group one youth did not want to
participate because he felt like he was taking too many groups at the facility. After some
encouragement from staff, the youth engaged in a role play activity and did very well. All youth
participated in the lesson and there were no behavior management issues.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and confirmed there have been no gaps in service delivery. Even
though there was turnover of facilitators, it did not adversely affect the T4C groups during the last
twelve months. There was facilitator turnover but the co-facilitator or another trained facilitator will
assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed twelve months of internal fidelity monitoring reports. Documentation confirmed staff
members who performed fidelity monitoring were trained by a qualified trainer to teach the primary
service. The program internal fidelity monitoring was conducted for all the facilitators once a
month within the last twelve months. Internal fidelity checklists are utilized to capture specifics of
the primary service. On the checklist, there is a corrective action follow-up component which
incorporated any applicable recommendations identified during fidelity monitoring. These
documents were maintained in a Thinking for a Change binder along with the T4C sign-in sheets.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Reviewed documentation for the last twelve months, verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. The
program has not had any applicable issues requiring corrective actions. The completed fidelity
monitoring forms recorded the observer’s feedback and recommendations, and ideas for
improvement in the delivery of the primary service Thinking for a Chance (T4C).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of two performance evaluation documents confirmed the director of case management
was acknowledged on the annual performance evaluation in delivering the primary service
Thinking for a Change (T4C) and conducting fidelity monitoring reviews. This included feedback
and recommendations of the facilitator’s proficiency in delivering the primary service. The clinical
director conducted fidelity monitoring reviews, but was not acknowledged on the annual review.
The transition coordinator and case manager were recently hired and did not have an annual
evaluation at the time of review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 15 weeks for this type of service. Of the 15 youth in the
sample, 100% (15 of 15) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 45 contact hours for this type of service. Of the 15 youth in the
sample, 67% (10 of 15) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 2 youth
High = 5 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 5 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (12 of 15) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 33% (5 of 15) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 17 25 68%
Totals 88 100 88%
This SPEP report evaluates Thinking for a Change (T4C), an intervention delivered at Palm
Beach Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 15 total youth sampled, all
included dosage with end dates in the EBS Module. Of those youth with correct dosage, all
received at least the recommended weeks of service. Youth in the sample completed between 15
and 31 weeks of service, with an average of 26 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, all included dosage in the EBS Module. Of those youth with correct dosage, 10 received
at least the recommended hours of service. Youth in the sample completed between 27 and 45
hours of service, with an average of 42 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Thinking for a Change


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Fairgrounds Road
Palm Beach, Florida 33411
Primary Service: Aggression Replacement Training (ART)
SPEP Review Date(s): January 9‐12, 2018
Florida Department of Juvenile Justice
Report Date(s): January 16, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1130
Provider Name: Sequel Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 78
Review Date(s): January 9-12, 2018 Lead Reviewer Code: 149
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The SPEP review encompasses twelve months of SPEP records, and examinations. Aggression
Replacement Training (ART) is a cognitive behavioral intervention program to help children, and
adolescents improve social skill competence, moral reasoning, manage anger, and reduce
aggressive behavior. ART specifically targets chronically aggressive children, and adolescents
ages twelve to seventeen. The program completed ten cohorts of Aggression Replacement
Training. An eleventh group was started during the annual review. ART sessions occur on
Mondays, Tuesdays, and Thursdays for approximately one hour each day. Youth are chosen
based on their needs as assessed by the RPACT, and Youth Needs Assessment Summary. The
program maintains all ART related documents in special binders which includes training records,
sign-sheets, program schedule, and internal fidelity monitoring sheets.
The program’s documentation reflected the program had eight staff who have received formal
training from a qualified trainer to become group facilitators in Aggression Replacement Training
(ART). Three facilitators were providing the primary service to the youth.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the evidenced-based curriculum of Aggression Replacement Training (ART)
curriculum. ART consists of 10 weeks (30 sessions) of intervention training, and is divided into
three components: social skills training, anger-control training, and training in moral reasoning.
Clients attend a one-hour session in each of these components three times a week. Incremental
learning, reinforcement techniques, and guided group discussions enhance skill acquisition and
reinforce the lessons in the curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Specialist was able to observe a group during the annual review period.
The group was held in a classroom, and had one facilitator. The group consisted of eight youth in
attendance. The facilitator followed along a script, but also used her own words to describe the
material to the youth. Most of the youth were engaged the full time, one youth had to be re-
directed to participate, but he did eventually choose to participate at a minimal level in the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Technical Assistance Specialist reviewed documentation, and confirmed there have been no gaps
in service delivery, nor turnover of facilitators which adversely affected ART groups during the last
twelve months. In the event of facilitator turnover, the co-facilitator or another trained facilitator will
assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Technical Assistance Specialist reviewed twelve months of internal fidelity monitoring reports.
Documentation confirmed staff members who performed fidelity monitoring were trained by a
qualified trainer to teach the primary service. The program's internal fidelity monitoring was
conducted for all the facilitators once a month within the last twelve months. Internal fidelity
checklists are utilized to capture specifics of the primary service. On the checklist, there is a
corrective action follow-up component which incorporated any applicable recommendations
identified during fidelity monitoring. These documents were maintained in an ART binder.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A reviewed documentation for the last twelve months, and verified the program has a process for
corrective actions based on the results of the internal fidelity monitoring. The program has not had
any applicable issues requiring corrective actions during the review period. The completed fidelity
monitoring forms recorded the observer’s feedback, recommendations, and ideas for
improvement in the delivery of the primary service Aggression Replacement Training (ART).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of performance evaluation documents confirmed the facilitators were acknowledged on
the annual performance evaluation in delivering the primary service Aggression Replacement
Training (ART). This included feedback and recommendations of the facilitator’s proficiency in
delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 10 weeks for this type of service. Of the 35 youth in the
sample, 46% (16 of 35) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 35 youth in the
sample, 100% (35 of 35) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 11 youth
High = 19 youth
Total Youth in Sample = 35 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 54%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 19 youth
Total Youth in Sample = 35 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (31 of 35) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 54% (19 of 35) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 92 100 92%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Palm
Beach Youth Academy.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Insitute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 Services (15 weeks; 45
hours).
The program earned 4 points for Amount of Service: Duration. Of the 35 total youth sampled, only
33 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 9
and 13 weeks of service, with an average of 10 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Contact Hours. Of the 35 total youth
sampled, all 35 included dosage in the EBS Module. Of those youth, all 35 received at least the
recommended hours of service. Youth in the sample completed between 30 and 45 hours of
service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Fairgrounds Road
Palm Beach, Florida 33411
Primary Service: Impact of Crime
SPEP Review Date(s): January 9‐12, 2018
Florida Department of Juvenile Justice
Report Date(s): January 16, 2016
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1130
Provider Name: Sequel Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 78
Review Date(s): January 9-12, 2018 Lead Reviewer Code: 149
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Impact of Crime is based on the belief crime is more than just a legal definition, but rather crime
affects the victim, the offender, their families, and the community. The harm caused brings with it
the moral responsibility to all involved. The main focus of Impact of Crime is to help offenders
understand the harm they have created, and then assisting them in taking personal accountability
for their actions. Together, the victim, offender, and the community arrive at a viable solution to
the repair the harm caused by the offender. Once the harm is repaired, the offender is
subsequently reintegrated back into the community. Facilitators are encouraged to bring in victim
impact speakers throughout the course of the curriculum.
The SPEP review encompasses twelve (12) months of SPEP records, and examinations.
According to documentation provided by the program, the facilitators of the primary service were
trained by a qualified trainer who was certified as a master trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes Impact of Crime: Addressing the Harm to Victims and the Community. The
curriculum was updated in 2009. The revised Impact of Crime is a victim impact/restorative justice
curriculum consisting of 7 Chapters taking approximately 24 sessions to facilitate.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
During the annual review, Technical Assistance Specialist observed a group utilizing the primary
service, Impact of Crime. The group was held on a module which was converted into an Art
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Gallery for the youth. The group consisted of eight youth, and was facilitated by a recreational
therapist who was trained in the primary service. The group was working in chapter four. The
facilitator read from the script, but also stopped, and asked youth their opinions of the statement
he had just read. The discussions lead to some youth expressing amblivance about their
previously held opinions. The group lasted approximately one hour, and the youth were engaged
during the time.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
In reviewing documentation, and informal interview with Clinical Director, no facilitator turnover
occurred. There were no gaps in service delivery of the primary service during the review period.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided documentation of internal fidelity monitoring which took place monthly.
The monitoring was performed by a trained facilitator, and spaces were provided for observational
feedback, facilitator’s ideas for improvement, and corrective actions, if needed
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a corrective action process for facilitators who need skill improvement. During
the review period, none of the internal fidelity monitoring forms documented the need for
corrective action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the two employee’s evaluations who are facilitating the primary service revealed that
there is a specific section where the employee is rated on their performance in delivering the
primary service. It scores the employee on their effectiveness as a facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 27 youth in the
sample, 74% (20 of 27) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 27 youth in the
sample, 74% (20 of 27) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 10 youth
High = 13 youth
Total Youth in Sample = 27 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 13 youth
Total Youth in Sample = 27 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (24 of 27) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 48% (13 of 27) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 75 85 88%
This SPEP report evaluates Impact of Crime, an intervention delivered at Palm Beach Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 27 total youth sampled,all
27 included dosage with end dates in the EBS Module. Of those youth, 20 received at least the
recommended weeks of service. Youth in the sample completed between 2 and 20 weeks of
service, with an average of 15 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 27 total youth
sampled, all 27 included dosage in the EBS Module. Of those youth, 20 received at least the
recommended hours of service. Youth in the sample completed between 2 and 43 hours of
service, with an average of 27 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Fairgrounds Road
Palm Beach, Florida 33411
Primary Service: Thinking for a Change
SPEP Review Date(s): January 9‐12, 2018
Florida Department of Juvenile Justice
Report Date(s): January 16. 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Palm Beach Youth Academy MQI Program Code: 1130
Provider Name: Sequel Contract Number: 10341
Location: Palm Beach County / Circuit: 15 Number of Beds: 78
Review Date(s): January 9-12, 2018 Lead Reviewer Code: 149
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve months of
SPEP records. The program completed four cohorts, and has two ongoing cohorts of Thinking for
a Change (T4C). The staff is utilizing the 4.0 curriculum. Groups were conducted as closed
groups, meaning once a group commenced, no new youth can be added to the group. Youth are
selected for the primary service based on their needs, as assessed by the R-PACT and Youth
Needs Assessment Summary. The program maintained all T4C related documents in a special
binder inclusive of all training records, sign-in sheets, and internal fidelity monitoring sheets. The
program had eight facilitators who had received the appropriate training to facilitate the primary
service, T4C.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provides the evidence based curriculum Thinking for a Change (T4C) 4.0 to
qualifying youth admitted to the program. Youth are selected for participation in the T4C group
based on their needs assessment and R-PACT.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical Assistance Specialist observed a group while on site during the annual review. The
facilitators shared the lesson plan with the observer prior to the group beginning. The group was
beginning lesson 17, Stop and Think. The group was co-facilitated by two trained case
managers, and had six youth in attendance. The facilitators followed the script. They also
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
stopped, and gave context in their own words. This allowed for the youth to ask questions, and
to discuss the topic. All youth participated, and were engaged during the lesson.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Technical Assistance Specialist reviewed documentation, and interviewed the Clinical Director. It
was confirmed there was no facilitator turnover during the last twelve months.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of twelve months of fidelity monitoring records was conducted. Documentation revealed
staff members who performed fidelity monitoring were trained by a qualified trainer to teach the
primary service. Fidelity monitoring was conducted at least once per month within the last twelve
months. Internal fidelity monitoring checklists were utilized to capture specifics of the primary
service. On the checklists, there was a corrective action follow-up component which incorporated
applicable recommendations identified during fidelity monitoring. These documents were
maintained in a Standardized Program Evaluation Protocol (SPEP) binder.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Technical Assistance Specialist conducted a review of previous twelve months fidelity monitoring
documentation. The review verified the program has a process for the application of corrective
actions based upon the results of fidelity monitoring conducted. The program had no issues
requiring corrective actions. The completed fidelity monitoring forms recorded the observer's
feedback and recommendations for improvement in the delivery of the primary service Thinking
for a Change (T4C).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Technical Assistance Specialist reviewed the annual reviews of staff who were actually facilitating
the primary service during the review period. All three performance evaluations had sections
noting the employee was a facilitator of the primary service, and rated their skills in delivering the
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 15 weeks for this type of service. Of the 20 youth in the
sample, 40% (8 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 20 youth in the
sample, 55% (11 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 5 youth
High = 11 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 55%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 11 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (16 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 55% (11 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 83 100 83%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Palm Beach
Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 20 total youth sampled, all
20 included dosage with end dates in the EBS Module. Of those youth, 8 received at least the
recommended weeks of service. Youth in the sample completed between 5 and 31 weeks of
service, with an average of 16 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, all 20 included dosage in the EBS Module. Of those youth, 11 received at least the
recommended hours of service. Youth in the sample completed between 18 and 52 hours of
service, with an average of 42 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the processes in place at the time of this review
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9