facility_name: Miami Youth Academy

summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 South West 84th Street
Miami, Florida 33173
Primary Service: The Council for Boys and Young Men
SPEP Review Date(s): August 11-14, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/6/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Miami Youth Academy QI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 11-14, 2015 Lead Reviewer Code: 107
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 6 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is The Council for Boys and Young Men. The program was
awarded 25 points because the Primary Service is identified as a Group 4 Service. The specific
Sub-Component Service Type identified is Group Counseling. The Primary Service was identified
as this type of service as it focuses on psychological or interpersonal problems or issues faced by
an individual and involves a group of youths interacting with each other
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program has six staff members who have each received
formal training from a qualified trainer to become a group facilitator in the primary service The
Council for Boys and Young Men. Each of the six staff members received training to be a
facilitator of this primary service in March of 2014. One staff member has facilitated each group
session and cycle since the last annual compliance review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes The Council for Boys and Young Men curriculum guide of, "Living a Legacy:
A Rite of Passage" for ages 14-18, for all applicable youth. The curriculum contained the order of
the lessons to be delivered, lesson plans for each session, delivery guidelines, and a script to
follow. The facilitator guide has lesson plans with instructions for conducting each group session.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The team observed a session during the review. Observations and interviews with the facilitator
during the annual compliance review supported the primary service The Council for Boys and
Young Men was delivered utilizing the curriculum guide. A copy of the lesson, “Week Three
Theme: Strength Through Diversity,” was provided to the annual compliance review team member
prior to the group session. Observations during the session found the facilitator adhered to the
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
lesson plan and the facilitator did not read the script verbatim during the activity, rather utilized
their own words to convey the meaning of the topic. The instructor taught the lesson and
redirected youth when necessary.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and interviews with the designated mental health clinician authority
(DMHCA) and regional clinical director confirmed there were no gaps in service delivery and no
turnover of facilitators since the primary service The Council for Boys and Young Men began in
December of 2014.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed documentation reflected the designated mental health clinician authority (DMHCA)
conducted fidelity monitoring and was trained by a qualified trainer in the primary service. Fidelity
monitoring was conducted in all applicable months for each facilitator of the primary service by the
DMHCA observing the facilitator’s group for the entire group session. Internal fidelity adherence
checklists are utilized monthly by the fidelity monitors to capture specifics of the primary service.
On the checklist, there is a corrective action follow-up component which incorporated any
recommendations identified during fidelity monitoring. These signed documents are maintained in
a Standardized Program Evaluation Protocol (SPEP) binder along with The Council for Boys and
Young Men attendance sign-in sheets. Fidelity checks are reviewed by the clinical director and
are taken to the next scheduled supervision meeting for discussion with the clinical team.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the regional clinical director found a process for qualified staff to develop a
corrective action based on the fidelity monitoring observations, if applicable. The facilitator’s
signature on the fidelity monitoring checklist document acknowledged any constructive feedback
given. The program has a practice in place to monitor this corrective action during the following
fidelity monitoring session. Reviewed fidelity monitoring documentation indicated formal corrective
action was not required.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of facilitator annual performance evaluations found an entry for evaluation of the staff’s
performance in delivering the primary service The Council for Boys and Young Men. The
performance evaluation comments on primary service delivery are derived from fidelity monitoring
observations, coaching sessions, and any disciplinary action related to the staff member's
provision of primary service curriculum. These included an evaluation of the facilitator’s skill and
quality of work and also comments on any strengths or weaknesses in which the facilitator should
be made aware.
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 9 youth
Total Youth in Sample = 20 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (18 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (9 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 73 95 77%
This SPEP report evaluates The Council for Boys and Young Men, an intervention delivered at
Miami Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 24 weeks. Of the 20 total youth sampled, only 19 had dosage recorded
in EBS with an end date. Of 19 youth with dosage, 0 received at least the recommended weeks of
service. Youth in the sample completed between 11 and 18 weeks of service, with an average of
12.3 weeks.
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 40 hours. Of the 20 total youth sampled, 0 received at least
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
the recommended hours of service. Youth in the sample completed between 9 and 13 hours of
service, with an average of 10.3 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Miami Youth Academy can continue to maintain their SPEP Quality of Service Delivery score by
continuing to follow the procedures in place at the time of this review.
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

report_type: SPEP

program_name_raw: Miami Youth Academy - TrueCore

program_status_note: Closed

service_type: The Council for Boys and Young Men


state: FL

summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 South West 84th Street
Miami, Florida 33173
Primary Service: Impact of Crime: Assessing the Harm to Victims and the
Community
SPEP Review Date(s): August 11-14, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/6/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Miami Youth Academy QI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 11-14, 2015 Lead Reviewer Code: 107
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 1 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program currently has one staff member trained to
facilitate Impact of Crime (IOC) and previously had one other IOC facilitator until June of 2015
when the staff member separated from the program. Each staff member received formal training
in August of 2014 from a qualified trainer to become a group facilitator in the primary service IOC.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provides the Department’s Impact of Crime (IOC) 2.0 curriculum, a practice with
demonstrated effectiveness, to all applicable youth admitted to the program. An interview with the
designated mental health clinician authority (DMHCA) confirmed that the facilitators use the IOC
manual to conduct each group. The manual contains the order of the lessons to be delivered,
lesson plans for each session, and delivery guidelines for the facilitator. Each group session has
an outline, worksheets, a guided group discussion activity, and a conclusion/wrap-up discussion
about the material covered.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An IOC group was observed on August 27, 2015, following the MQI review. The facilitator
followed the chapter, as outlined in the facilitator guide, and completed all activities as required.
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Impact of Crime curriculum is designed for delivery to groups of 8 to 12 youths. There were only
three youth in this group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and interviews with the regional compliance manager, regional clinical
director, and designated mental health clinician authority (DMHCA) confirmed there was a gap in
service delivery from June 28, 2015, through July 13, 2015, due to facilitator turnover. At the time
of this annual compliance review, the DMHCA was the only trained facilitator of the primary
service Impact of Crime (IOC). An additional interview with the facility administrator and regional
clinical director confirmed that the program presently has one staff member receiving IOC training
and has two more staff scheduled to receive IOC.
The program can earn 2 points if there is no facilitator turnover and no gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed documentation validated the designated mental health clinician authority (DMHCA)
conducts fidelity monitoring and was trained by a qualified trainer. Staff whom are trained in the
primary service Impact of Crime (IOC) may conduct internal fidelity monitoring at the program by
observing another facilitator’s group for the entire group session at least once per month. Due to
facilitator turnover, fidelity monitoring was not completed for the month of July 2015. The program
utilizes an IOC-specific fidelity adherence checklist. On the checklist, there is a corrective action
follow-up component, which may incorporate any applicable recommendations identified during
fidelity monitoring. These signed documents are maintained in a SPEP binder along with the IOC
attendance sign-in sheets. Fidelity checks are reviewed by the clinical director and are taken to
the next scheduled supervision meeting for discussion with the clinical team.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
An interview with the regional clinical director found a process for qualified staff to develop a
corrective action based on the fidelity monitoring observations, if applicable. Reviewed
documentation found fidelity monitoring was conducted by the designated mental health clinician
authority (DMHCA), whom was trained by a qualified trainer. There were consistent critiques and
recommendations, if applicable, where the fidelity monitor provided constructive feedback to the
facilitator. The facilitator’s signature on the fidelity monitoring checklist document acknowledged
the constructive feedback and improvements. The program has a practice in place to monitor
corrective action during the following fidelity monitoring session. Reviewed fidelity monitoring
documentation indicated corrective action was not required.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of facilitator annual performance evaluations found an entry for evaluation of the staff’s
performance in delivering the primary service Impact of Crime (IOC). The performance evaluation
comments on primary service delivery are derived from fidelity monitoring observations, coaching
sessions, and any disciplinary action that was provided to the staff member regarding the
provision of primary service curricula. These included an evaluation of the facilitator’s skill and
quality of work and also comments on any strengths or weaknesses.
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 11 youth in the
sample, 18% (2 of 11) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 11 youth in the
sample, 64% (7 of 11) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 3 youth
High = 6 youth
Total Youth in Sample = 11 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 55%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 11 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (10 of 11) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 55% (6 of 11) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 69 85 81%
This SPEP report evaluates Impact of Crime: Addressing the Harm to Victims and the Community
(IOC), an intervention delivered at Miami Youth Academy.
The program scored High for Quality of Service Delivery. This score can be optimized by
minimizing turnover and increasing the frequency of internal fidelity monitoring (see section 2d-e
above). Impact of Crime should be delivered to groups of 8 to 12 youth.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 16 weeks. Of the 11 total youth sampled, 0 received at least the
recommended weeks of service. The minimum number of weeks completed for a youth was 5,
and the maximum number of weeks completed for a youth was 20. The average number of weeks
for this sample was 9.9.
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
The program earned 6 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 24 hours. Of the 11 total youth sampled, 7 received at least
the recommended hours of service. The minimum number of hours completed for a youth was 12,
and the maximum number of hours completed for a youth was 26. The average number of hours
for this sample was 23.2.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at Miami Youth Academy.
The program itself has no control over youths' C-PACT risk level because the scored assessment
was administered prior to the youths' admission.
RECOMMENDATION(S):
Miami Youth Academy can optimize their SPEP Quality of Service Delivery score by minimizing
turnover and increasing the frequency of internal fidelity monitoring.
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

source: DJJ

service_type: Impact of Crime


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 South West 84th Street
Miami, Florida 33173
Primary Service: Criminal Conduct and Substance Abuse Treatment for
Adolescents: Pathways to Self-Discovery and Change
SPEP Review Date(s): August 11-14, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/6/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Miami Youth Academy QI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 11-14, 2015 Lead Reviewer Code: 107
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Criminal Conduct and Substance Abuse Treatment for
Adolescents: Pathways to Self-Discovery and Change. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program has four staff members who have each received
formal training from a qualified trainer to become a group facilitator in the primary service Criminal
Conduct and Substance Abuse Treatment for Adolescents: Pathways to Self-Discovery and
Change. Each of the four staff received training in March 2014. One staff member has facilitated
each group session and cycle since the last annual compliance review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provides the Criminal Conduct and Substance Abuse Treatment for Adolescents:
Pathways to Self-Discovery and Change Second Edition curricula as a primary service in the
program. The curriculum has fifteen chapters and contained lesson plans with instructions for
conducting each group session, the order of the lessons to be delivered, delivery guidelines, and
a script to follow.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The team observed a session during the review. Observations and interviews with the facilitator
during the annual compliance review supported the primary service Criminal Conduct and
Substance Abuse Treatment for Adolescents: Pathways to Self-Discovery and Change was
delivered utilizing the appropriate curriculum. A copy of session eight, “Identifying Triggers and
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Your High-Risk Situations,” of the lesson plan was provided to the annual compliance review team
member prior to the group session. The lesson plan for the session was effective and the handout
work papers were followed. Observations during the session found the facilitator adhered to the
lesson plan and the facilitator did not read the script verbatim during the activity, rather utilized
their own words to convey the meaning of the topic of discussion. The instructor taught the lesson
and redirected youth when necessary.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and interviews with the designated mental health clinician authority
(DMHCA) and regional clinical director confirmed there has been no gap in service delivery and
no turnover of facilitators since the last annual compliance review.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed documentation reflected the designated mental health clinician authority (DMHCA)
conducted fidelity monitoring and was trained by a qualified trainer to facilitate the primary service.
Fidelity monitoring was conducted each month for each facilitator of the primary service by having
the DMHCA observe the facilitator’s group for the entire group session. Internal fidelity adherence
checklists are utilized monthly and include a corrective action follow-up component, which
incorporated any applicable recommendations identified during fidelity monitoring. These signed
documents are maintained in a Standardized Program Evaluation Protocol (SPEP) binder along
with the Pathways to Self-Discovery and Change attendance sign-in sheets. Fidelity checks are
reviewed by the clinical director and are taken to the next scheduled supervision meeting for
discussion with the clinical team.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the DMHCA and regional clinical director found a process for qualified staff to
develop a corrective action based on the fidelity monitoring observations. There were consistent
critiques and recommendations, when applicable, where the fidelity monitor provided constructive
feedback to the facilitator. The facilitator’s signature on the fidelity monitoring checklist document
acknowledged the constructive feedback and improvements. The program has a practice in place
to monitor this corrective action during the following fidelity monitoring session. Reviewed fidelity
monitoring documentation indicated corrective action was not required.
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of facilitator annual performance evaluations found an entry for evaluation of the staff’s
performance in delivering the primary service Criminal Conduct and Substance Abuse Treatment
for Adolescents: Pathways to Self-Discovery and Change. The performance evaluation comments
on primary service delivery are derived from fidelity monitoring observations, coaching sessions,
and any disciplinary action that was provided to the staff member regarding the provision of
primary service curricula. These included an evaluation of the facilitator’s skill and quality of work
and also comments on any strengths or weaknesses.
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 15% (3 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 90%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 9 youth
Total Youth in Sample = 20 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 90% (18 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (9 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 73 95 77%
This SPEP report evaluates Criminal Conduct and Substance Abuse Treatment for Adolescents:
Pathways to Self-Discovery and Change, an intervention delivered at Miami Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 24 weeks. Of the 20 total youth sampled, 0 received at least the
recommended weeks of service. Youth in the sample completed between 16 and 22 weeks of
service, with an average of 17.5 weeks.
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 40 hours. Of the 20 total youth sampled, 3 received at least
the recommended hours of service. Youth in the sample completed between 32 and 43 hours of
service, with an average of 32.8 hours.
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Miami can continue to maintain their SPEP Quality of Service Delivery score by continuing to
follow the procedures in place at the time of this review.
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

service_type: Pathways to Self‐Discovery and Change


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 South West 84th Street
Miami, Florida 33173
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): August 11-14, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/6/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Miami Youth Academy QI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 11-14, 2015 Lead Reviewer Code: 107
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 1 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 7 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Reviewed documentation reflected the program has two staff members who have each received
formal training from a qualified trainer to become a group facilitator in the primary service Thinking
for a Change (T4C). Each of the staff were trained in April of 2014. One staff member facilitated
the two most recent T4C group cycles since the last annual compliance review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program currently uses the evidenced-based curriculum of Thinking for a Change (T4C) 3.1.
The curriculum provides a script and instructions for twenty-five lessons. The curriculum also
includes implementation guidelines and identifies the order the lessons delivered.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The team observed a session during the review. Observations and interviews with the facilitator
during the annual compliance review supported the primary service Thinking for a Change (T4C)
was delivered utilizing the T4C 3.1 curriculum. A copy of lesson twelve, “Social Skill: Making a
Complaint,” was provided to the annual compliance review team member prior to the group
session. Observations during the session found the facilitator adhered to the lesson plan and the
facilitator did not read the script verbatim during the activity, rather utilized their own words to
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
convey the meaning of the topic of discussion. The instructor taught the youth on the lesson and
redirected youth when necessary.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Reviewed documentation and interviews with the designated mental health clinician authority
(DMHCA) and regional clinical director confirmed there were no gaps in service delivery and no
turnover of facilitators since the primary service Thinking for a Change (T4C) began in December,
2014.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Reviewed documentation reflected the designated mental health clinician authority (DMHCA)
conducted fidelity monitoring and was trained by a qualified trainer to teach the primary service.
Fidelity monitoring was conducted in all applicable months for each facilitator of the primary
service by having the DMHCA, who observes the facilitator’s group for the entire group session at
least once per month while evaluating the facilitator. Internal fidelity adherence checklists are
utilized monthly by the fidelity monitors to capture specifics of the primary service. On the
checklist there is a corrective action follow-up component, which incorporated any applicable
recommendations identified during fidelity monitoring. These signed documents are maintained in
a Standardized Program Evaluation Protocol (SPEP) binder along with the Thinking for a Change
attendance sign-in sheets. Fidelity checks are reviewed by the clinical director and are taken to
the next scheduled supervision meeting for discussion with the clinical team.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the DMHCA and regional clinical director found a process for qualified staff to
develop a corrective action based on the fidelity monitoring observations. Reviewed
documentation found fidelity monitoring was conducted by trained staff. There were consistent
critiques and recommendations, if applicable, where the fidelity monitor provided constructive
feedback to the facilitator. The facilitator’s signature on the fidelity monitoring checklist
acknowledged the constructive feedback and improvements. The program has a practice in place
to monitor this corrective action during the following fidelity monitoring session. Reviewed fidelity
monitoring documentation indicated formal corrective action was not required.
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of facilitator annual performance evaluations found an entry for evaluation of the staff’s
performance in delivering the primary service Thinking for a Change (T4C). The performance
evaluation comments on primary service delivery are derived from fidelity monitoring
observations, coaching sessions, and any disciplinary action provided to the staff member
regarding the provision of primary service curricula. These included an evaluation of the
facilitator’s skill and quality of work and also comments on any strengths or weaknesses.
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 19 youth in the
sample, 11% (2 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 19 youth in the
sample, 0% (0 of 19) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 8 youth
High = 8 youth
Total Youth in Sample = 19 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (17 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (8 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 78 100 78%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Miami Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 15 weeks. Of the 19 total youth sampled, only 17 had dosage recorded
in EBS with end dates. Of 17 youth with dosage, 2 received at least the recommended weeks of
service. Youth in the sample completed between 5.5 and 19 weeks of service, with an average of
11.3 weeks.
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 45 hours. Of the 19 total youth sampled, 0 received at least
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
the recommended hours of service. Youth in the sample completed between 9 and 33 hours of
service, with an average of 19 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Miami Youth Academy can continue to maintain their SPEP Quality of Service Delivery score by
continuing to follow the procedures in place at the time of this review.
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

service_type: Thinking for a Change


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 SW 84th Street
Miami, Florida 33173
Primary Service: The Council For Boys and Young Men (Boys Council)
SPEP Review Date(s): August 16‐19, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/30/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Miami Youth Academy MQI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 16-19, 2016 Lead Reviewer Code: 149
Persons Interviewed
Program Director Corporate QI/QA staff 3 # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee 3 # Clinical Staff 8 # Other (listed by title): Youth
DMHCA or designee # Healthcare Staff Care Specialist
Documents Reviewed
Written Protocol/Manual Logbooks 8 # Personnel Records
Fidelity Monitoring Documents Program Schedules 8 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is The Council For Boys and Young Men. The program was
awarded 25 points because the Primary Service is identified as a Group 4 Service. The specific
Sub-Component Service Type identified is Group Counseling. The Primary Service was identified
as this type of service as it focuses on psychological or interpersonal problems or issues faced by
an individual and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected a training protocol and five staff members received formal
training from a qualified trainer to become group facilitators in Boys Council. All five staff members
were trained in March, 2014. Three therapists, the assistant facility administrator (AFA), and one
youth specialist were trained in March, 2014. Only one staff member has facilitated groups during
the last twelve months. An interview with the clinical director confirmed, by means of their policy,
each new facilitator must successfully conduct two cycles with a co-facilitator before they can
facilitate a group session on their own.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for the team to
reference during the annual compliance review. The program currently uses The Council for Boys
and Young Men’s Program. The curriculum provides a script and instructions for thirty lessons.
The curriculum also includes implementation guidelines, identifies the order the lessons should be
delivered, and includes detailed instructions for the delivery of each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The annual compliance review team observed a session of Boys Council during the annual
compliance review. The facilitator provided a copy of the lesson plan for the observer from the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
program manual. The facilitator followed the lesson according to the manual, did not read the
lesson verbatim, used his own words to convey the meaning of the discussed topic, and did so in
an easy to follow manner. The facilitator adhered to the lesson plan in every aspect.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and interviews with the clinical director and corporate clinical
services director confirmed there have been no turnover adversely affecting the youth during the
past twelve months. The program’s policy regarding turnover is the anticipated turnover is
addressed by ensuring any potentially affected groups are co-facilitated at the outset. There were
no gaps in services.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The annual compliance review team reviewed twelve months of internal fidelity monitoring reports.
The program had an internal process to conduct internal fidelity monitoring specific to Boys
Council; their process indicated the clinical director performed all Boys Council fidelity monitoring.
The clinical director was also trained in Boys Council. Fidelity monitoring was conducted all
applicable months the facilitator of the primary service.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the clinical director and corporate clinical services director verified the program
has a process for corrective actions based on fidelity monitoring conducted. The program uses
internal fidelity monitoring/corrective action checklists specific to Boys Council. Each fidelity
monitoring checklist includes a section for feedback and recommendations for facilitator
improvement. Each completed checklist included feedback indicating the facilitator did not require
any corrective action, or included recommendations for future improvement. The following
month’s checklist included comments the previous months’ recommendations were applied and
demonstrated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of five facilitator’s employee performance evaluations found an entry for evaluation of the
staff’s performance in delivering Boys Council. These included comments on any strengths or
weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 11 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 61%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 11 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (18 of 18) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 61% (11 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 75 95 79%
This SPEP report evaluates The Council For Boys and Young Men (Boys Council), an
intervention delivered at Miami Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 18 total youth sampled, only
8 included dosage with end dates in the EBS Module. Of those youth with correct dosage, none
received at least the recommended weeks of service. Youth in the sample completed between 10
and 21 weeks of service, with an average of 15 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 9 and 21 hours of service, with an average of 13 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Miami Youth Academy
G4S Youth Services, LLC
(Contract Provider)
10855 SW 84th Street
Miami, Florida 33173
Primary Service: Impact of Crime
SPEP Review Date(s): August 16‐19, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/30/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Miami Youth Academy MQI Program Code: 1289
Provider Name: G4S Youth Services, LLC Contract Number: 10000
Location: Miami-Dade County / Circuit: 11 Number of Beds: 24
Review Date(s): August 16-19, 2016 Lead Reviewer Code: 149
Persons Interviewed
Program Director Corporate QI/QA staff 3 # Program Supervisors
DJJ Monitor 2 # Case Managers 10 # Youth
DHA or designee 3 # Clinical Staff 8 # Other (listed by title): Youth
DMHCA or designee # Healthcare Staff Care Specialist
Documents Reviewed
Written Protocol/Manual Logbooks 8 # Personnel Records
Fidelity Monitoring Documents Program Schedules 8 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected the program had a training protocol and four staff
members received formal training from a qualified trainer to become a group facilitator in Impact
of Crime (IOC). The clinical director was trained in December, 2015 and two youth specialists
were also trained in December, 2015. A youth specialist was trained August, 2014. Two staff
members have facilitated groups during the last twelve months.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for the annual
compliance review team to reference during the review. The program currently uses the Impact of
Crime (IOC) curriculum. The curriculum provides a script and instructions for seven chapters with
twenty-four lessons. The curriculum also includes implementation guidelines, identifies the order
how each chapter and lessons within should be delivered, and includes detailed instructions for
the delivery of each lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
d. Facilitator Turnover Basic Score: 2 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and interviews with the clinical director and corporate clinical
services director confirmed there has been no turnover adversely affecting the youth during the
past twelve months. The program’s policy and procedures regarding turnover is the anticipated
turnover is addressed by ensuring any potentially affected groups are co-facilitated at the outset.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The team reviewed twelve months of internal fidelity monitoring reports. The program had an
internal process to conduct internal fidelity monitoring specific to the Impact of Crime (IOC); their
process was the clinical director and case manager director performed all IOC fidelity monitoring.
The clinical director and case manager director were also trained in IOC. One staff member
received internal fidelity monitoring during the months September through December, 2015.
Another staff received fidelity monitoring from January through July, 2016. Fidelity monitoring was
conducted during all other applicable months for each facilitator of the primary service.
BasicScore: 0Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the corporate clinical services director verified the program has a process for
corrective actions based on fidelity monitoring conducted. The program uses internal fidelity
monitoring/corrective action checklists specific to Impact of Crime (IOC). Each fidelity monitoring
checklist includes a section for feedback and recommendations for facilitator improvement. Each
completed checklist included feedback the facilitator did not require any corrective action, or
included recommendations for future improvement. The reviewed month’s checklist did not
include comments indicating the previous months’ recommendations were applied and
demonstrated. The fidelity monitoring checklist from September 24, 2015, did not match the
lesson listed on the sign-in sheet for September 24, 2015. Although the sign-in sheets reflect a
staff member was regularly facilitating IOC groups on Tuesdays and Thursdays, the case
manager director mentions this staff member was asked at the last minute to facilitate group and
he “accepted the challenge.” During the November, 2015 review, the case manager director
recommended for facilitator to review group norms, expectations, and consequences. He also
explained on his report the group needed “structure”; these recommendations were not addressed
during the December 2015 review. However, what was addressed during the following review is
the facilitator failed to review group norms again, which was a recommendation from the previous
report. Recommendations from fidelity monitoring review performed on March, 2016, were not
confirmed as having been applied and demonstrated by facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 1 point by ensuring that any needed corrective action is implemented and
reflected in subsequent internal fidelity monitoring reports.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of four facilitator’s performance evaluations found no entry for evaluation of the staff’s
performance in delivering IOC. These did not include comments on any strengths or weaknesses
of which the facilitator should be aware.
The program can earn 1 point if performance evaluations specifically identify the facilitators'
proficiency in delivering IOC.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 13 youth in the
sample, 0% (0 of 13) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 13 youth in the
sample, 92% (12 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 4 youth
High = 8 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 92%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 92% (8 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 63 85 74%
This SPEP report evaluates Impact of Crime, an intervention delivered at Miami Youth Academy.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring that identified corrective action is remedied and documented in internal fidelity
monitoring reports, and that facilitators' performance evaluations include an assessment of their
performance in the delivery of IOC.
The program earned 0 points for Amount of Service: Duration. Of the 13 total youth sampled,
none received at least the recommended weeks of service. Youth in the sample completed
between 11 and 12 weeks of service.
The program earned 8 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, 12 received at least the recommended hours of service. All but one youth completed 24
hours of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Miami Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring that
identified corrective action is remedied and documented in internal fidelity monitoring reports, and
that facilitators' performance evaluations include an assessment of their performance in the
delivery of IOC.
Miami Youth Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10