state: FL

facility_name: Orlando Intensive Youth Academy

summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orlando Intensive Youth Academy
G4S Youth Services, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Impact of Crime
SPEP Review Date(s): May 31‐ June 3, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Orlando Intensive Youth Academy QI Program Code: 1166
Provider Name: G4S Youth Services, LLC Contract Number: 10145
Location: Orange County / Circuit: 9 Number of Beds: 16
Review Date(s): May 31- June 3, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation reviewed showed the program has a formal training protocol for training to be
provided through DJJ from a qualified trainer. The program has four staff that are trained in
Impact of Crime and actively conducting groups. The staff are two Transition Services Managers
and two Case Managers. All four staff have facilitated this intervention within the last twelve
months.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator has a copy of the manual for Impact of Crime and the reviewer was provided a
copy during this review. There are no separate workbooks for the youth. The manual contains
lessons plans with instructions for implementation and discussion topics.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed during the review. The facilitators provided the
reviewer a copy of the lesson. Nine youth were in attendance for this session. The session
included a basic overview and introduction to the curriculum and opened the dialogue of possible
events and activities that align with the restorative justice focus. The facilitators were in
compliance with the goal and purpose of the first lesson.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
During the twelve-month review period, the program had one facilitator who resigned from the
program. The cycle for this facilitator was complete before his resignation. There is no evidence
of gaps in service because the program already had alternative staff in place. The program plans
to keep additional staff trained in the event of future turnovers.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided documentation of twelve months of internal fidelity checks in support of the
program’s process for internal fidelity. The checks are being performed by the Transition Services
Manager (TSM). Because the TSM is also a part of the rotation of facilitators, the TSM does not
receive internal fidelity.
The program can earn 2 points if every facilitator receives monthly fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the DCM and a review of documentation confirmed that the program has a
process for corrective action in place as needed to support fidelity. During the review period of
twelve months, the program had one minor issue needing correction related to scheduled time for
the service. The issue was resolved and there were no other issues requiring correction.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the annual performance evaluations for the four staff trained in this intervention,
verified that all are evaluated on meeting the proficiency standards for facilitating IOC.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 5 youth in the
sample, 0% (0 of 5) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 5 youth in the
sample, 100% (5 of 5) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 3 youth
High = 2 youth
Total Youth in Sample = 5 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 40%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 2 youth
Total Youth in Sample = 5 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (5 of 5) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 40% (2 of 5) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 75 85 88%
This SPEP report evaluates Impact of Crime, an intervention delivered at Orlando Intensive Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 5 total youth sampled, none
received at least the recommended weeks of service. Youth in the sample completed between 9
and 13 weeks of service, with an average of 11 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 5 total youth
sampled, all 5 received at least the recommended hours of service. All youth completed 24 hours
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Orlando Intensive Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring all facilitators receive monthly fidelity monitoring.
Orlando Intensive Youth Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Orlando Intensive Youth Academy - TrueCore

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orlando Intensive Youth Academy
G4S Youth Services, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Thinking for a Change
SPEP Review Date(s): May 31‐ June 3, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Orlando Intensive Youth Academy QI Program Code: 1166
Provider Name: G4S Youth Services, LLC Contract Number: 10145
Location: Orange County / Circuit: 9 Number of Beds: 16
Review Date(s): May 31- June 3, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation showed the program adheres to a formal training protocol offered
through a DJJ-approved qualified trainer. The training takes place over a four-day period requiring
32 hours of skilled instruction. The program has seven facilitators trained to conduct this
intervention (three serving in the position of Case Manager, two Transition Services, one
Administrative Assistant, and one Director of Case Management). During the review period six
staff have actively facilitated this intervention.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator is provided with a physical copy of the Thinking for a Change manual during the
four-day training. There are no individual workbooks for the youth, the manual includes
reproducible handouts and worksheets for the majority of the 25 lessons. The detailed manual is
divided into three sections of concentration: Cognitive Self-Change, Social Skills, and Problem
Solving. The T4C manual is specific to delivery, script, and outline.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed during the review. The lesson title was Stop
and Think. The facilitators provided a copy of lesson from the manual for the observer to follow
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
along. The facilitators followed the lesson according to the manual. The use of visual aids and
youth role plays was also used according to the curriculum.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documents and interview with the Director of Case Management supported that the
program has not had any facilitator turnover in the past twelve months. In the event of turnover,
the program has several trained facilitators that are prepared to rotate into position until other staff
are trained to replace the missing facilitators. During this review period, there are no issues with
gaps in service.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Twelve months of documentation of internal fidelity monitoring reports were reviewed. The
program’s internal fidelity process for this curriculum is completed monthly by the Director of Case
Management. A total of six different facilitators conducted this group and received fidelity
monitoring over the last twelve months.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the Director of Case Management verified that the program has a process for
corrective action in place as needed to support the fidelity monitoring process. After reviewing the
documentation for the twelve-month review period, there were no issues requiring corrective
action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The review of annual performance evaluations for the, verified that all staff members are
evaluated specifically on T4C and are meeting the proficiency standard for facilitating this primary
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 15 weeks for this type of service. Of the 6 youth in the
sample, 50% (3 of 6) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 6 youth in the
sample, 0% (0 of 6) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 0 youth
High = 5 youth
Total Youth in Sample = 6 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 83%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 6 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (6 of 6) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 83% (5 of 6) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 84 100 84%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Orlando Intensive
Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 6 total youth sampled, 3
received at least the recommended weeks of service. Youth in the sample completed between 12
and 21 weeks of service, with an average of 17 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 6 total youth sampled,
none received at least the recommended hours of service. Youth in the sample completed
between 25 and 41 hours of service, with an average of 33 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Orlando Intensive Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the same practices in place at the time of this review.
Orlando Intensive Youth Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Thinking for a Change


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orlando Intensive Youth Academy
G4S Youth Services, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Teen Relationships Workbook
SPEP Review Date(s): May 31‐ June 3, 2016
Florida Department of Juvenile Justice
Report Date(s): 5/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Orlando Intensive Youth Academy QI Program Code: 1166
Provider Name: G4S Youth Services, LLC Contract Number: 10145
Location: Orange County / Circuit: 9 Number of Beds: 16
Review Date(s): May 31- June 3, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff 1 # Other (listed by title): Regional
DMHCA or designee # Healthcare Staff Clinical Director
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Teen Relationships. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation reviewed showed that the program has a training protocol. Training for this
ervices is informal and is included in the on the job staff training for each member of the
therapeutic team. This training is provided by either the Clinical Director or the Regional Clinical
Director. During the review period the program had seven facilitators trained to conduct this
intervention, six therapists and one clinical director. Over the past twelve months, five therapists
have actively facilitated this intervention.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a physical copy of the manual for the facilitator to adhere to when running this
intervention. There are no individual workbooks for the youth, but each therapist has a digital copy
of the manual for immediate access to information. The detailed manual has 68 reproducible
worksheets to encourage youth in developing healthy relationships.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session was observed during the review. The facilitator provided a copy of the lesson so that
the observer could follow along. Each youth was given a worksheet that coincided with the topic
of the lesson. The facilitator stayed true to the fidelity of the lesson and created an atmosphere for
interaction and learning for the participants.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The clinical director and the regional clinical director interviewed to discuss facilitator turnover and
gaps in service. The program also provided documentation to confirm there was no facilitator
turnover or gaps in service in the last twelve months. One facilitator retired in April 2016, but this
did not create any staffing issues for this intervention. In the event of a facilitator turnover the
program has two facilitators available to rotate into the position so that there is no gap in service
to the youth.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided documentation of twelve months of internal fidelity checks in support of
their in-house process for intervention accountability. The checks are performed by the clinical
director for all facilitators on a monthly basis. The clinical director is also trained in the
intervention. The reviewer found no discrepancies with the internal fidelity documentation.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the clinical director and regional clinical director verified the program has a
process for corrective action in place as needed to support the fidelity monitoring process. After
reviewing the documentation for twelve months, there was no need for correction action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of annual performance evaluations for the seven members of the therapeutic staff
verified that all are assessed on their proficiency in facilitating this primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 13 youth in the
sample, 85% (11 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 13 youth in the
sample, 15% (2 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 3 youth
High = 8 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 62%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (12 of 13) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 62% (8 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 81 95 85%
This SPEP report evaluates Teen Relationships Workbook, an intervention delivered at Orlando
Intensive Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 13 total youth sampled, 11
received at least the recommended weeks of service. Youth in the sample completed between 11
and 47 weeks of service, with an average of 31 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, 2 received at least the recommended hours of service. Youth in the sample completed
between 23 and 44 hours of service, with an average of 31 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Orlando Intensive Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the processes in place at the time of this review.
Orlando Intensive Youth Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Teen Relationships


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
G4S Youth Services, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): May 2 ‐ 5, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/28/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Orange Youth Academy/Orlando Intensive Youth Academy
MQI Program Code:
1127/1166
Provider Name: G4S Youth Services, LLC Contract Number: 10145
Location: Orange County / Circuit: 9 Number of Beds: 40
Review Date(s): May 2 - 5, 2017 Lead Reviewer Code: 097
Persons Interviewed
Program Director Corporate QI/QA staff 2 # Program Supervisors
DJJ Monitor 1 # Case Managers 3 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 7 # Personnel Records
Fidelity Monitoring Documents Program Schedules 7 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 7 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had six facilitators trained in the Impact of Crime curriculum. Out of those, four
actually facilitated groups in the last twelve months. A review of the facilitator training
documentation showed the facilitators all received training in the Impact of Crime curriculum from
a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
All youth admitted at Orange Youth Academy/Orlando Intensive Youth Academy are assessed by
LMHC staff as to need for services. If they are in need of the Impact of Crime curriculum, they are
then placed in the group. The program has a written manual detailing the primary service. The
program uses the Impact of Crime “Addressing the Harm” manual. The manual is detailed. It
includes a script, implementation guidelines, gives the order the lessons are to be delivered in,
and includes detailed instructions for delivery of each lesson. Youth are chosen based on their
needs as assessed by the RPACT and Youth Needs Assessment Summary. The program
maintains a SPEP binder with all of their fidelity checks, sign-in sheets, training records,
evaluations, and all other information pertinent to the intervention.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The primary service Impact of Crime is normally held on Mondays and Fridays. An observation of
the class was completed on Friday 05/05/17. A copy of the lesson was provided to the observer.
The group was lesson 3-Martin Luther King’s “ I have a Dream” speech and discussing how to
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
handle conflict. The facilitator adhered to the manual and put the lesson in his own words which
made it very easy for the youth to understand.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has completed four cycles of Impact of Crime during the review period. The program
started a cycle of Impact of Crime on 03/03/17 and that group was ongoing at the time of the
review. The reviewed documentation confirmed there had been facilitator turnover in the last 12
months, however, this did not cause any gap in service. The program has a policy for facilitator
turnover. If a facilitator is going to be leaving the program or will be on leave, they explain to the
youth why another facilitator will be taking over the class. Another qualified facilitator would then
either take over the class or if necessary, a staff member would be brought from another facility to
conduct the class.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Fidelity monitoring was provided by formally trained staff for the facilitators. The fidelity monitors
have received certification in the primary service and fidelity monitoring. The program utilizes
fidelity adherence checklists to document the monitoring. The fidelity monitoring documentation
was reviewed and it was found that monitoring was completed each month for the review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. The program utilizes an internal fidelity monitoring
corrective action checklist to specify if a facilitator is in need of corrective action or improvement.
There was no corrective action for the facilitators in the last 12 months.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A performance evaluation was completed for all facilitators who facilitated the intervention during
the review period. Of the evaluations reviewed all were was found to contain an entry specific to
the staff member who facilitated the Impact of Crime intervention and their ability to demonstrate
an appropriate level of skills when delivering the primary curriculum.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 18 youth in the
sample, 0% (0 of 18) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 18 youth in the
sample, 94% (17 of 18) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 7 youth
High = 8 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (17 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 44% (8 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Orange Youth
Academy/Orlando Intensive Youth Academy. This report combines scores for both programs.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 18 total youth sampled,
none received at least the recommended weeks of service. Youth in the sample completed
between 7 and 12 weeks of service, with an average of 11.3 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, 17 received at least the recommended hours of service. Youth in the sample completed
between 17 and 24 hours of service, with an average of 23 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Quality of
Service Delivery score by minimizing facilitator turnover.
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Amount of
Service score by ensuring that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orange Youth Academy/Orlando Intensive Youth Academy
G4S Youth Services, LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): May 2 ‐ 5, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/28/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Orange Youth Academy/Orlando Intensive Youth Academy
MQI Program Code:
1127/1166
Provider Name: G4S Youth Services, LLC Contract Number: 10145
Location: Manatee County / Circuit: 9 Number of Beds: 40
Review Date(s): May 2 - 5, 2017 Lead Reviewer Code: 097
Persons Interviewed
Program Director Corporate QI/QA staff 2 # Program Supervisors
DJJ Monitor 1 # Case Managers 3 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 7 # Personnel Records
Fidelity Monitoring Documents Program Schedules 7 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 7 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking For A Change (T4C). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had ten facilitators trained in the Thinking for a Change curriculum. Out of those,
four actually facilitated groups in the last twelve months. A review of the facilitator training
documentation confirmed the facilitators all received training in the Thinking for a Change
curriculum from a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
All youth admitted at Orange Youth Academy/Orlando Intensive Youth Academy are assessed by
LMHC staff as to need for services. If they are in need of Thinking for a Change, they are then
enrolled and placed in the class. The program has a written manual detailing the primary service
and provided a copy for the team for reference during the review. The program uses the Thinking
for a Change manual by NIC, Version 3.1. The curriculum provides a script and instructions for
twenty five lessons, includes implementation guidelines, identifies the order the lessons should be
delivered, and includes detailed instructions for the delivery of each lesson.
This is considered a closed group, meaning once the group begins, no new youth can be added
to the group. The program conducts one T4C cycle at a time. New admissions are placed on a
waiting list and are then enrolled in the next class. Group sessions take place on Mondays and
Fridays, each one an hour in length and ranges from one to ten youth at a time per group. Youth
are chosen based on their needs as assessed by the RPACT and Youth Needs Assessment
Summary. The program maintains a SPEP binder with all of their fidelity checks, sign-in sheets,
training records, facilitator evaluations, and all other information pertinent to the intervention.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The primary service Thinking for a Change is held on Mondays and Fridays. An observation of the
class was completed on 05/05/17. The lesson was Lesson 24- Problem Solving. The group
discussed problem solving skills and how to develop them and learn to use them daily. The
facilitator followed the manual and she did not read the lesson verbatim. She used her own words
to convey the lesson to the youth which made it easier for them to understand.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has conducted two cycles of Thinking for a Change (T4C) during the review period.
The program started a cycle of Thinking for a Change (T4C) on 12/05/16 and that cycle will end
on 05/05/17.The reviewed documentation confirmed there had been facilitator turnover in the last
12 months, however, this was not disruptive to the services provided to the youth. The program
has a policy for facilitator turnover. If a facilitator is going to be leaving the program or will be on
leave, they explain to the youth why another facilitator will be taking over the class. Another
qualified facilitator would then either take over the class or if necessary, a staff member would be
brought from another facility to facilitate the class.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Fidelity monitoring was provided by the formally trained staff for the facilitators. The fidelity
monitors have received certification in the primary service and fidelity monitoring. The program
utilizes fidelity adherence checklists to document the monitoring. The fidelity monitoring
documentation was reviewed and it was found that monitoring was completed each month for the
review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process by which corrective action is acknowledged and demonstrated based
on the fidelity monitoring of the primary service. The program utilizes an internal fidelity monitoring
corrective action checklist to specify if a facilitator is in need of corrective action or improvement.
No corrective action was taken for the facilitators for Thinking for a Change in the last 12 months.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A performance evaluation was completed for all facilitators who facilitated the intervention during
the review period. Of the evaluations reviewed, all were found to contain an entry specific to the
staff member who facilitated the Thinking for a Change intervention and their ability to
demonstrate an appropriate level of skills when delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 15 weeks for this type of service. Of the 15 youth in the
sample, 73% (11 of 15) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 15 youth in the
sample, 0% (0 of 15) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 4 youth
High = 11 youth
Total Youth in Sample = 15 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 73%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 11 youth
Total Youth in Sample = 15 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (15 of 15) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 73% (11 of 15) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 86 100 86%
This SPEP report evaluates Thinking For A Change (T4C) and Thinking For A Change Aftercare,
an intervention delivered at Orange Youth Academy/Orlando Intensive Youth Academy. This
report combines scores for both programs.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 15 total youth sampled, only
12 included dosage with end dates in the EBS Module. Of those youth, 11 received at least the
recommended weeks of service. Youth in the sample completed between 3 and 20.5 weeks of
service, with an average of 18 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 15 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 5 and 42 hours of service, with an average of 35 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Orange Youth Academy/Orlando Intensive Youth Academy can optimize their SPEP Amount of
Service score by ensuring that data is entered correctly in the EBS Module and by ensuring youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Orlando Intensive Youth Academy
Truecore Behavioral LLC
(Contract Provider)
3150 39th Street
Orlando, Florida 32839
Primary Service: Impact of Crime
SPEP Review Date(s): 04/17/18‐04/20/18
Florida Department of Juvenile Justice
Report Date(s): 04/20/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Program Name: Orange Youth Academy/Orlando Intensive MQI Program Code: 10145
Provider Name: Truecore Behavorial LLC Contract Number: R1227
Location: Orange County / Circuit: 9 Number of Beds: 41/16
Review Date(s): April 17-20, 2018 Lead Reviewer Code: 148
Persons Interviewed
Program Director 2 # Case Managers 5#
DJJ Monitor 2 # Clinical Staff 5 # Youth
DHA or designee 2 # Healthcare Staff # Other (listed by title):
DMHCA or designee 5 # Program Supervisors
Corporate QI/QA staff
Documents Reviewed
Written Protocol/Manual Logbooks 7 # Personnel Records
Fidelity Monitoring Documents Program Schedules 7 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There were five staff trained in the Impact of Crime (IOC) curriculum - one AFA, two TSM's, one
Director Case Management, one Administrative Assistant. Two of the facilitators had been
consistently facilitating this group during the review period. One of the staff left employment.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided to this reviewer was Impact of Crime. The manual contains a script for the
facilitator for the groups. It provides lessons and includes implementation guidelines, identifies the
order the lessons should be delivered in and includes detailed instructions for the delivery of each
lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This curriculum was not able to be observed; therefore this indicator rates as non-applicable. The
Impact of Crime group was not running during the review period as the facility had switched to
running Thinking for a Change.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
There has been recent facilitator turnover prior to this review. However, a review of
documentation provided, and an interview with the Director of Case Management (DCM),
confirmed the turnover itself did not adversely affect the services to the youth and did not occur
during a cohort. The program, in the case of turnover, would use administrative or other trained
staff to facilitate the curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Fidelity monitoring was completed for all months in the review period by the Director of Case
Management. There was no missing fidelity monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the Director of Case Management (DCM) verified that the program has a
process for corrective actions based on the fidelity monitoring conducted. The program uses
internal fidelity monitoring/corrective action checklists specific to Impact of Crime. A review of
documentation provided and an interview with the DCM verified that there was no corrective
action needed nor taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The performance evaluation of all trained facilitators found entries on all evaluations of the staff's
performance in the delivery of the Impact of Crime curriculum. These included comments on any
strengths or weaknesses of which the facilitator should be aware.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 8 youth in the
sample, 0% (0 of 8) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 8 youth in the
sample, 75% (6 of 8) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 2 youth
Moderate-High = 2 youth
High = 4 youth
Total Youth in Sample = 8 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 8 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (8 of 8) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 50% (4 of 8) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 25 25 100%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime, an intervention delivered at Orlando Intensive Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 8 total youth sampled, only
6 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 11
and 14 weeks of service, with an average of 12 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 8 total youth sampled,
all 8 included dosage in the EBS Module. Of those youth, 6 received at least the recommended
hours of service. Youth in the sample completed between 18 and 25 hours of service, with an
average of 23 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Orlando Intensive Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Orlando Intensive Youth Academy can optimize their SPEP Amount of Service score by ensuring
that dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9