summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33315
Primary Service: Aggression Replacement Training
SPEP Review Date(s): September 11‐14, 2018
Florida Department of Juvenile Justice
Report Date(s): September 17, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses SPEP records since
the last Monitoring and Quality Improvement (MQI) review, conducted on January 12, 2018. The
program’s documentation reflected the program had a training protocol and there were five staff
who received formal training from a qualified trainer to become a group facilitator in Aggression
Replacement Training (ART), during the review period. One therapist was trained on August 28,
2015, two therapists were trained on May 15, 2015, one clinical director was trained on March 6,
2015 and an assistant clinical director was trained on May 19, 2015.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilized the Aggression Replacement Training (ART) curriculum. The
program has a manual for the delivery of services and a workbook for the students to use. A copy
of the manual was provided for the team to reference during the review. ART is a multimodal
psychoeducational intervention designed to alter the behavior of chronically aggressive
adolescents and young children. The goal of ART is to improve social skill competence, anger
control, and moral reasoning. The curriculum consists of a ten week, thirty-hour intervention
administered to groups of eight to twelve youth three times a week for one hour each session
which was followed by the program on a consistent basis during the review period.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
An observation of an Aggression Replacement Training (ART) group was conducted on
September 11, 2018. The lesson covered was Anger Control Training (Reminders). There were
eight youth participating in group. The facilitator was prepared for group and was knowledgeable
of the material presented. The facilitator correctly introduced the anger control chain and
demonstrated the skills associated with controlling anger. All youth participated in the group role
play activity and were able to demonstrate the anger steps. There were no behavior issues during
group. The classroom was used to present the group which was a comfortable setting.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The Standardized Program Evaluation Protocol (SPEP) review encompasses SPEP records since
the last Monitoring and Quality Improvement (MQI) review conducted on January 12, 2018. The
program completed two cohorts of Aggression Replacement Training (ART) and one cohort was
in progress during the review. The first cohort started on January 4, 2018 and was completed on
March 14, 2018. Eight youth started group, one was removed for behavior issues and seven
successfully completed. The second cohort started on January 29, 2018 and was completed on
April 12, 2018. Eight youth started group, two youth were removed due to behavior issues and six
successfully completed. The ART group in progress at the time of the review started on August
20, 2018 with eight youth participating. Youth are chosen to participate in ART based on their
needs as assessed by the Residential Positive Achievement Change Tool (R-PACT) and a Mental
Health/Substance Abuse evaluation. Groups are scheduled three times a week for an hour each
session.
The reviewed documentation confirmed there was an approximate three-month gap in service
delivery of ART since the last review. The program submitted a request to the Department’s
contract manager to replace ART as a primary service to another anger management curriculum
and were waiting for a response. Technical assistance was provided to the program On August 9,
2018 and the technical assistance specialist recommended the program should continue running
ART groups until they receive notification from the Department. The program started a new ART
group on August 20, 2018 and recently received notification from the Department to continue
providing ART to the youth. In the event of facilitator turnover another trained staff will assume the
role of the facilitator for the group.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The reviewed internal fidelity monitoring reports since the MQI review conducted on January 12,
2018, confirmed staff members who performed fidelity monitoring were trained by a qualified
trainer to teach the primary service of Aggression Replacement Training (ART). The program
internal fidelity monitoring was conducted on a monthly basis for all of the facilitators. Internal
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
fidelity checklists are utilized to capture specifics of the primary service. On the checklist, there is
a corrective action follow-up component which incorporated any applicable recommendations
identified during fidelity monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The reviewed documentation verified the program has a process for the application of corrective
actions based on the results of the fidelity monitoring conducted. According the clinical director
and the reviewed documentation, the program did not note any applicable issues requiring
corrective actions. The completed fidelity monitoring forms recorded the observer’s feedback and
recommendations, and ideas for improvement in the delivery of the primary service Aggression
Replacement Training (ART).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four annual performance evaluations of the staff who facilitated Aggression
Replacement Training (ART) during the last twelve months were acknowledged and rated as
outstanding in delivering the curriculum and/or conducting internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 69 youth in the
sample, 84% (58 of 69) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 69 youth in the
sample, 96% (66 of 69) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 26 youth
High = 36 youth
Total Youth in Sample = 69 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 52%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 36 youth
Total Youth in Sample = 69 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (63 of 69) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 52% (36 of 69) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 94 100 94%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Palm
Beach Youth Academy.
71 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Duration. Of the 69 total youth sampled, only
68 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 58
received at least the recommended weeks of service. Youth in the sample completed between 4
and 15 weeks of service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 69 total youth
sampled, all 69 included dosage in the EBS Module. Of those youth, 66 received at least the
recommended hours of service. Youth in the sample completed between 5 and 45 hours of
service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring there are no unnecessary gaps in service.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Palm Beach Youth Academy

summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33411
Primary Service: Impact of Crime
SPEP Review Date(s): September 11‐14, 2018
Florida Department of Juvenile Justice
Report Date(s): September 17, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100 %
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program has a training protocol and there were five
staff who received formal training from a qualified trainer to become a group facilitator in Impact of
Crime (IOC) during the last twelve months. At the time of the review, the facility had two
recreational therapists trained on July 22, 2016, one director of case management trained on
December 12, 2011, one clinical director trained on February 20, 2015 and one assistant program
director of operations was trained on April 11, 2014.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitators utilized the Impact of Crime (IOC), Addressing the Harm curriculum.
The program has a manual for the delivery of services and a workbook for the students to use. A
copy of the manual and workbook was provided for the team to reference during the review. The
curriculum provides a script and instructions for seven chapters which requires a minimum of
thirty-two contact hours to facilitate. The curriculum includes implementation guidelines, identifies
the order the lessons are to be delivered, and provides detailed instructions for the delivery of
each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of an Impact of Crime (IOC) group was conducted on September 12, 2018. The
group discussed Chapter 6 "Managing Conflict" with a focus on Hate Crimes. The group was held
in the art room, which was a pleasant environment and conducive to learning. The facilitator
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
reviewed the chapter objectives and words to know. The facilitator was prepared for group,
demonstrated knowledge of the material and displayed an objective point of view. The topic
discussed can be a hot button topic and brought out a lot of emotions and a heated discussion
between the youth. The facilitator was able to control the conversation, deescalate the heated
discussions and remind youth to respect everyone's opinions. All youth participated in the group
discussion and there were no behavior issues.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The Standardized Program Evaluation Protocol (SPEP) review encompasses SPEP records since
the last Monitoring and Quality Improvement (MQI) review conducted on January 12, 2018. The
program completed two cohorts of Impact of Crime (IOC) and two cohorts were in progress during
the review period. The first cohort of IOC started on November 20, 2017 and ended on May 9,
2018. There were eleven youth attending group, one youth successfully completed group, ten
youth were removed from the group due to release from the program and/or behavior issues.
Technical assistance was provided to the program due to a significant percentage of youth not
completing groups. On August 9, 2018, the clinical director and technical assistance specialist
met at the program and discussed ways to ensure youth who start groups can successfully
complete groups. The second cohort started on January 3, 2018 and was completed on May 16,
2018. Twelve youth started group and eight youth successfully completed. The two cohorts that
were in progress started on June 13, 2018 and June 11, 2018. Youth are chosen to participate in
IOC based on their needs as assessed by the Residential Positive Achievement Change Tool (R-
PACT) and a Mental Health/Substance Abuse evaluation. Groups are held twice a week for an
hour each session.
The reviewed documentation confirmed there has not been a gap in service which adversely
affected the IOC groups during the last twelve months. In the event of facilitator turnover, another
trained staff will assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of internal fidelity monitoring reports since the last review validated staff members who
performed internal fidelity monitoring were trained by a qualified trainer. Internal fidelity monitoring
was consistently conducted on a monthly basis for each group and each facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation since the last review verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. An
interview with the clinical director confirmed the program has not noted any applicable issues
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
requiring corrective actions. The completed internal fidelity monitoring forms recorded the
observer’s feedback and corrective action recommendations, and ideas for improvement in the
delivery of the primary service of Impact of Crime (IOC).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four annual performance evaluations of the staff who facilitated Impact of Crime (IOC)
and/or conducted internal fidelity monitoring were acknowledged for having good to outstanding
skills in delivering the curriculum and/or conducting internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 72 youth in the
sample, 36% (26 of 72) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 72 youth in the
sample, 39% (28 of 72) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 29 youth
High = 35 youth
Total Youth in Sample = 72 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 35 youth
Total Youth in Sample = 72 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (65 of 72) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 49% (35 of 72) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at Palm Beach Youth
Academy.
79 youth completed the program during the review period. Of those youth, 72 had not previously
received this intervention in another setting.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 72 total youth sampled, only
42 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 26
received at least the recommended weeks of service. Youth in the sample completed between 4
and 25 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Contact Hours. Of the 72 total youth
sampled, only 43 included dosage in the EBS Module. Of those youth with correct dosage, 28
received at least the recommended hours of service. Youth in the sample completed between 2
and 41 hours of service, with an average of 27 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 81%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Palm Beach Youth Academy - Sequel

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33411
Primary Service: Thinking for a Change
SPEP Review Date(s): September 11‐14, 2018
Florida Department of Juvenile Justice
Report Date(s): September 17, 2018
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 1
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses SPEP records since
the last Monitoring and Quality Improvement (MQI) review conducted on January 12, 2018. The
program is currently utilizing the Thinking for a Change (T4C) 4.0 curriculum. The program’s
documentation and the Department's Learning Management System (SkillPro) reflected the
program had seven staff trained by a certified trainer to facilitate and/or provide internal fidelity
monitoring during the review period. The director of case management (DCM) and the assistant
program director of operations were trained on November 21, 2014, two case managers were
trained on June 9, 2017, one case manager and one transition manager were trained on January
13, 2017, and the clinical director was trained on March 27, 2015.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the evidenced-based curriculum of Thinking for a Change (T4C) 4.0 to
qualifying youth admitted to the program. A copy of the manual and workbook were provided for
the MQI team to reference during the review. T4C is comprised of 25 lessons that build upon each
other, and contains appendices that can be used to craft an aftercare program to meet ongoing
cognitive behavioral needs of the group. Not all lessons can be completed in one session, so a
typical delivery cycle may take thirty sessions. Sessions should last between one and two hours.
Ideally, the curriculum is delivered two times per week, with a minimum recommended dosage of
once per week and a maximum of three times per week. Participants must be granted time to
complete mandatory homework between each lesson.
T4C is intended for groups of eight to twelve and should be delivered only by trained facilitators.
Due to its integrated structure, T4C is a closed group, meaning members need to start at the
beginning of a cycle, and may not join the group mid-stream. The curriculum provides a script and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
instructions for twenty-five lessons. The curriculum includes implementation guidelines, identifies
the order the lessons are to be delivered, and detailed instructions for each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of the Thinking for a Change (T4C) group was conducted on September 11, 2018.
The lesson taught was "Introduction to Problem Solving" The facilitator reviewed the homework
assigned at the end of the previous lesson before the observer arrived. The facilitator utilized a
computer and projector to display the "Bender" clip from the movie "The Breakfast Club". There
was a good discussion and role play scenarios with youth and facilitator regarding the movie clip.
The facilitator followed the curriculum and explained the lesson to the youth in plain language. All
youth participated in the lesson and there were no behavior issues. The group was held in the art
room which was a relaxing and comfortable environment to conduct group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The SPEP review included records since the last Monitoring and Quality Improvement (MQI)
review conducted on January 12, 2018. The program completed four cohorts of Thinking for a
Change (T4C) and three cohorts were in progress during the review period. The first cohort of
T4C started on November 14, 2017 and ended on February 22, 2018. Eight youth started in the
group, but six youth successfully completed because two were removed due to behavior issues.
The next three cohorts started on March 15, 2018 and were completed on June 12, 2018. All
three cohorts in progress during the review, started on July 17, 2018. Youth are chosen to
participate in T4C based on their needs as assessed by the Residential Positive Achievement
Change Tool (R-PACT) and a Mental Health/Substance Abuse evaluation. Groups are scheduled
twice a week for an hour and a half each session.
The reviewed documentation confirmed there has not been a gap in service which adversely
affected the T4C groups during the last twelve months. In the event of facilitator turnover, another
trained staff will assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of internal fidelity monitoring reports starting from the last Monitoring and Quality
Improvement (MQI) review conducted on January 12, 2018, validated staff members who
performed internal fidelity monitoring were trained by a qualified trainer. The documents provided
indicated internal fidelity monitoring was conducted on a monthly basis for all groups and
facilitators.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation since the last review verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. An
interview with the clinical director confirmed the program has not noted any applicable issues
requiring corrective actions. The completed internal fidelity monitoring forms recorded the
observer’s feedback and recommendations, and ideas for improvement in the delivery of the
primary service Thinking for a Change (T4C).
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of six annual performance evaluations of the staff which facilitated Thinking for a Change
(T4C) and/or conducted internal fidelity monitoring indicated staff were rated as very good or
outstanding in delivering the curriculum and/or providing internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 77 youth in the
sample, 32% (25 of 77) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 77 youth in the
sample, 55% (42 of 77) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 31 youth
High = 38 youth
Total Youth in Sample = 77 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 38 youth
Total Youth in Sample = 77 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (70 of 77) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 49% (38 of 77) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 25%
Totals 84 100 84%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Palm Beach
Youth Academy.
79 youth completed the program during the review period. Of those youth, 77 had not previously
received this intervention in another setting.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 77 total youth sampled, only
48 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 25
received at least the recommended weeks of service. Youth in the sample completed between 7
and 26 weeks of service, with an average of 15 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 77 total youth
sampled, only 48 included dosage in the EBS Module. Of those youth with correct dosage, 42
received at least the recommended hours of service. Youth in the sample completed between 24
and 82 hours of service, with an average of 47 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 86%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33411
Primary Service: Aggression Replacement Training
SPEP Review Date(s): October 15‐18, 2019
Florida Department of Juvenile Justice
Report Date(s): October 21, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve (12) months
of records examinations. The program’s documentation reflected the program had a training
protocol and there were four staff who received formal training from a qualified trainer to become
a group facilitator in Aggression Replacement Training, during the review period. Two therapists
were trained on September 20, 2019, one therapist was trained on May 15, 2015, one clinical
director was trained on March 6, 2015.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilized the Aggression Replacement Training (ART) curriculum. The
program has a manual for the delivery of services and a workbook for the students to use. A copy
of the manual was provided for the team to reference during the review. ART is a multimodal
psychoeducational intervention designed to alter the behavior of chronically aggressive
adolescents and young children. The goal of ART is to improve social skill competence, anger
control, and moral reasoning. The curriculum consists of a ten week, thirty-hour intervention
administered to groups of eight to twelve youth three times a week for one hour each session
which was followed by the program on a consistent basis during the review period.
Youth are chosen to participate in ART based on their needs as assessed by the Residential
Assessment for Youth (RAY) and a Mental Health/Substance Abuse evaluation. Groups are
scheduled three times a week for an hour each session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of an Aggression Replacement Training (ART) group was conducted on October
15, 2019. The topic covered was Skillstreaming: Dealing with Accusations. There were eight
youth participating in group. The facilitator was prepared for group and was knowledgeable of the
material presented. The facilitator reviewed the group rules and the skillstreaming steps
associated with the lesson. The facilitator modeled the skill and all youth participated in the group
role play activity. There were no behavior issues during group. The classroom was used to
present the group which was a comfortable setting.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of documentation confirmed there were no gaps in service delivery of Aggression
Replacement Training (ART) during the review period, nor was there any facilitator turnover. In
the event of facilitator turnover another trained staff will assume the role of the facilitator for the
group.
The program completed five cohorts of ART and two cohorts were in progress during the review.
The first cohort started on August 20, 2018 and was completed on October 25, 2018. Eight youth
completed successfully. The second cohort started on October 22, 2018 and was completed on
December 18, 2018. Eight youth successfully completed ART. The third cohort started on January
28, 2019 and completed on April 5, 2019. Eight youth successfully completed ART. The fourth
cohort started on April 9, 2019 and was completed on June 20, 2019 with nine youth successfully
completing ART. The fifth cohort started on April 18, 2019 and ended on June 27, 2019 with nine
youth successfully completing group. The ART groups in progress at the time of the review
started on August 5, 2019 with nine youth participating and on August 26, 2019 with eight youth
participating.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of internal fidelity monitoring reports confirmed staff members who performed fidelity
monitoring were trained by a qualified trainer to facilitate the primary service of Aggression
Replacement Training. The internal fidelity monitoring was conducted on a monthly basis for all
facilitators. Internal fidelity checklists are utilized to capture specifics of the primary service. On
the checklist, there is a corrective action follow-up component which incorporated any applicable
recommendations identified during fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation verified the program has a process for the application of corrective
actions based on the results of the fidelity monitoring conducted. According the clinical director
and the reviewed documentation, the program did not have any applicable issues requiring
corrective actions. The completed fidelity monitoring forms recorded the observer’s feedback and
recommendations, and ideas for improvement in the delivery of the primary service, Aggression
Replacement Training.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four annual performance evaluations of the staff who facilitated Aggression
Replacement Training during the last twelve months were acknowledged and rated as
outstanding in delivering the curriculum and/or conducting internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 10 weeks for this type of service. Of the 54 youth in the
sample, 81% (44 of 54) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 54 youth in the
sample, 98% (53 of 54) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 17 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 68%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 81%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 11 youth
Moderate-High = 15 youth
High = 18 youth
Total Youth in Sample = 54 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 33%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 18 youth
Total Youth in Sample = 54 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 81% (44 of 54) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 33% (18 of 54) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 17 25 68%
Totals 88 100 88%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at Palm
Beach Youth Academy.
83 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Duration. Of the 54 total youth sampled, all
54 included dosage with end dates in the EBS Module. Of those youth, 44 received at least the
recommended weeks of service. Youth in the sample completed between 9 and 11 weeks of
service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 54 total youth
sampled, all 54 included dosage in the EBS Module. Of those youth, 53 received at least the
recommended hours of service. Youth in the sample completed between 27 and 30 hours of
service, with an average of 30 hours.
The program was awarded 17 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 96%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Aggression Replacement Training


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33411
Primary Service: Impact of Crime
SPEP Review Date(s): October 15-18, 2019
Florida Department of Juvenile Justice
Report Date(s): October 21, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program had a training protocol and there were five
staff who received formal training from a qualified trainer to become a group facilitator in Impact of
Crime during the last twelve months. At the time of the review, the facility had two recreational
therapists trained on July 22, 2016, one director of case management trained on December 12,
2011, one clinical director trained on February 20, 2015 and one assistant program director of
operations was trained on April 11, 2014.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitators utilized the Impact of Crime, Addressing the Harm curriculum. The
program has a manual for the delivery of services and a workbook for the students to use. A copy
of the manual and workbook was provided for the team to reference during the review. The
curriculum provides a script and instructions for seven chapters which requires a minimum of
thirty-two contact hours to facilitate. The curriculum includes implementation guidelines, identifies
the order the lessons are to be delivered, and provides detailed instructions for the delivery of
each lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The primary service of Impact of Crime was not observed during the SPEP review due to a
scheduling conflict; therefore, the indicator is rated N/A.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve months of
records examinations. The sign-in sheets provided by the program verified there were four
cohorts of Impact of Crime (IOC) completed and two cohorts were in progress during the review
period. The first cohort of IOC started on November 19, 2018 and was completed on March 6,
2019. The second cohort started on November 19, 2018 and was completed on March 10, 2019.
The third cohort started on April 1, 2019 and was completed on July 17, 2019. The fourth cohort
started on April 3, 2019 and was completed on July 19, 2019. The cohorts in progress at the time
of the review started on August 5 and August 7, 2019. Youth are chosen to participate in IOC
based on their needs as assessed by the Residential Assessment for Youth (RAY) and a Mental
Health/Substance Abuse evaluation. Groups are held twice a week for an hour each session.
The reviewed documentation confirmed there has not been a gap in service which adversely
affected the IOC groups during the last twelve months. In the event of facilitator turnover, another
trained staff will assume the role of the facilitator for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of internal fidelity monitoring reports since the last review validated staff members who
performed internal fidelity monitoring were trained by a qualified trainer. Internal fidelity monitoring
was consistently conducted on a monthly basis for each group and each facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation since the last review verified the program has a process for the
application of corrective actions based on the results of the fidelity monitoring conducted. An
interview with the clinical director confirmed the program has not had any applicable issues
requiring corrective actions. The completed internal fidelity monitoring forms recorded the
observer’s feedback and corrective action recommendations, and ideas for improvement in the
delivery of the primary service of Impact of Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of two performance evaluations of the staff who facilitated Impact of Crime and/or
conducted internal fidelity monitoring were acknowledged for having good to outstanding skills in
delivering the curriculum and/or conducted internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 76 youth in the
sample, 64% (49 of 76) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 76 youth in the
sample, 67% (51 of 76) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 14 youth
Moderate-High = 21 youth
High = 28 youth
Total Youth in Sample = 76 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 37%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 28 youth
Total Youth in Sample = 76 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (63 of 76) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 37% (28 of 76) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 72 85 85%
This SPEP report evaluates Impact of Crime, an intervention delivered at Palm Beach Youth
Academy.
83 youth completed the program during the review period. Of those youth, 76 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 76 total youth sampled, only
51 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 49
received at least the recommended weeks of service. Youth in the sample completed between 2
and 18 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Contact Hours. Of the 76 total youth
sampled, only 52 included dosage in the EBS Module. Of those youth with correct dosage, 51
received at least the recommended hours of service. Youth in the sample completed between 1
and 34 hours of service, with an average of 32 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 91%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Palm Beach Youth Academy
Sequel Youth and Family Services
(Contract Provider)
9680 Weisman Way
Palm Beach, Florida 33411
Primary Service: Thinking for a Change
SPEP Review Date(s): October 15‐18, 2019
Florida Department of Juvenile Justice
Report Date(s): October 21, 2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The Standardized Program Evaluation Protocol (SPEP) review encompasses twelve (12) months
of records examinations. The program is currently utilizing the Thinking for a Change (T4C) 4.0
curriculum. The program’s documentation and the Department's Learning Management System
(SkillPro) reflected the program had ten (10) staff trained by a certified trainer to facilitate and/or
provide internal fidelity monitoring during the review period. The director of case management
(DCM) and the assistant program director of operations were trained on November 21, 2014, two
case managers were trained on June 9, 2017, one transition manager was trained on January 13,
2017, two case managers were trained on November 16, 2018, two case managers were trained
on June 14, 2019 and the clinical director was trained on March 27, 2015.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the evidenced-based curriculum of Thinking for a Change (T4C) 4.0 to
qualifying youth admitted to the program. A copy of the manual and workbook were provided for
the Monitoring and Quality Improvement team to reference during the review. T4C is comprised of
25 lessons that build upon each other, and contains appendices that can be used to craft an
aftercare program to meet ongoing cognitive behavioral needs of the group. Not all lessons can
be completed in one session, so a typical delivery cycle may take thirty sessions. Sessions should
last between one and two hours. Ideally, the curriculum is delivered two times per week, with a
minimum recommended dosage of once per week and a maximum of three times per week.
Participants must be granted time to complete mandatory homework between each lesson.
T4C is intended for groups of eight to twelve (12) and should be delivered only by trained
facilitators. Due to its integrated structure, T4C is a closed group, meaning members need to start
at the beginning of a cycle, and may not join the group mid-stream. The curriculum provides a
script and instructions for twenty-five lessons. The curriculum includes implementation guidelines,
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
identifies the order the lessons are to be delivered, and detailed instructions for each lesson.
Youth are chosen to participate in T4C based on their needs as assessed by the Residential
Assessment for Youth (RAY) and a Mental Health/Substance Abuse evaluation. Groups are
scheduled twice a week for an hour and a half each session.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of the Thinking for a Change (T4C) group was conducted on October 17, 2019.
The lesson taught was "Negotiating" The facilitators started with an icebreaker, reviewed the
group rules, main points of the previous chapter and homework.The facilitators gave out the
handouts for the lesson to the youth. They followed the curriculum verbatim. Affirmations were
used throughout the lesson. The facilitators were knowledgeable of the material presented and
comfortable facilitating group. The facilitators modeled the new skill and all youth participated in
the role play activity. The group was held in the classroom which was a comfortable environment
to conduct group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program completed seven cohorts of Thinking for a Change (T4C) and three cohorts were in
progress during the review period. Four cohorts of T4C started on December 11, 2018 and ended
on March 19, 2019. Six youth started in each group, and all youth participating successfully
completed. The next three cohorts started on April 9, 2019 and were completed on July 31, 2019.
However, most of the sign-in sheets for all three cohorts were missing. The program provided
sign-in sheets for group one dated May 1, May 13, May 20, May 27 and May 29, 2019. Sign-in
sheets for group two were dated May 15, May 22, June 3, June 6, and June 19, 2019. The sign-in
sheets provided for group three were dated May 13, May 22, May 29, June 3, and June 10, 2019.
The clinical director informed the technical assistance unit the sign-in sheets went missing after
they were entered in the Department's Evidence Based System (EBS). A review of the EBS
system verified the cohorts were entered. One cohort in progress during the review, started on
July 16, 2019 and two cohorts started on September 12, 2019.
The reviewed documentation confirmed there has not been a gap in service which adversely
affected the T4C groups during the last twelve (12) months. In the event of facilitator turnover,
another trained staff will assume the role of the facilitator for the group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of internal fidelity monitoring reports validated staff members who performed internal
fidelity monitoring were trained by a qualified trainer. The documents provided indicated internal
fidelity monitoring was conducted on a monthly basis for all groups and facilitators.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation verified the program has a process for the application of corrective
actions based on the results of the fidelity monitoring conducted. An interview with the clinical
director confirmed the program has not had any applicable issues requiring corrective actions.
The completed internal fidelity monitoring forms recorded the observer’s feedback and
recommendations, and ideas for improvement in the delivery of the primary service Thinking for a
Change.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of ten (10) annual performance evaluations of the staff which facilitated Thinking for a
Change and/or conducted internal fidelity monitoring indicated staff were rated as very good or
outstanding in delivering the curriculum and/or providing internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 15 weeks for this type of service. Of the 83 youth in the
sample, 46% (38 of 83) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 45 contact hours for this type of service. Of the 83 youth in the
sample, 63% (52 of 83) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 84%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 15 youth
Moderate-High = 24 youth
High = 31 youth
Total Youth in Sample = 83 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 37%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 31 youth
Total Youth in Sample = 83 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 84% (70 of 83) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 37% (31 of 83) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 85 100 85%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Palm Beach
Youth Academy.
83 youth completed the program during the review period. Of those youth, all 83 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 83 total youth sampled, only
50 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 38
received at least the recommended weeks of service. Youth in the sample completed between 6
and 22 weeks of service, with an average of 15 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Contact Hours. Of the 83 total youth
sampled, only 55 included dosage in the EBS Module. Of those youth with correct dosage, 52
received at least the recommended hours of service. Youth in the sample completed between 18
and 65 hours of service, with an average of 46 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 90%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Palm Beach Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Palm Beach Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9