state: FL

facility_name: Duval Academy

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Youth Academy
Sequel Youth Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Impact for Change (IOC)
SPEP Review Date(s): March 21‐24, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth services Contract Number: 10094
Location: Jacksonville County / Circuit: 4 Number of Beds: 28
Review Date(s): March 21-24, 2017 Lead Reviewer Code: 129
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 2 # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact for Change (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation reviewed showed evidence that the program has a formal training protocol
that is provided through DJJ from a qualified trainer. The program has two staff that are trained in
Impact of Crime and they are actively conducting groups. The Clinical Director has noted that two
new hires are presently scheduled to attend the next IOC training.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator has a copy of the manual for Impact of Crime and the MQI reviewer was provided
a copy during this review. There are no separate workbooks for the youth. The manual contains
lessons plans with basic instructions for implementation and discussions.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The MQI reviewer observed a session of Impact of Crime during the review. The facilitators
provided the MQI staff with a copy of the lesson. Eight youth were in attendance for this session.
The session included a highlight from the previous discussion and dialogue about how it feels to
be robbed. The youth were attentive and participated without being prompted. The facilitator was
in compliance with the goal and purpose of the lesson.
d. Facilitator Turnover Basic Score: 1 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There is no evidence of facilitator turnover. There are several gaps in service from March 2016 –
November 2016. The two trained facilitators were ready and available to conduct this intervention,
but the clinical director, at that time, did not prioritize this on the schedule. The new clinical
director has reorganized group interventions and will maintain better documentation and facilitator
accountability.
The program can earn 2 points by minimizing gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program provided documentation of three months of internal fidelity checks in support of the
program’s reorganization for internal fidelity. The checks are being performed by curriculum
trained staff with accountability from the clinical director.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the clinical director provided a clear plan for corrective action based on fidelity
monitoring. There were no corrective actions needed during the last three months.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
This MQI reviewer examined the staff evaluations for the trained facilitators of this intervention.
There is no mention of or evaluation of facilitator skills. This was discussed with the clinical
director who will adjust the forms to include facilitator skills for the upcoming evaluation period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of the delivery of IOC.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 4 youth in the
sample, 0% (0 of 4) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 4 youth in the
sample, 50% (2 of 4) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 4 youth
Total Youth in Sample = 4 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 100%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 4 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (4 of 4) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 100% (4 of 4) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 25 25 100%
Totals 59 85 69%
This SPEP report evaluates Impact for Crime (IOC), an intervention delivered at Duval Youth
Academy.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
minimizing gaps in service, by ensuring that internal fidelity monitoring is conducted at least
monthly for all facilitators, and by ensuring that facilitators' performance evaluations include an
assessment of the delivery of IOC.
The program earned 0 points for Amount of Service: Duration. Of the 4 total youth sampled, none
received at least the recommended weeks of service. Youth in the sample completed between 2
and 12 weeks of service, with an average of 8 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 4 points for Amount of Service: Contact Hours. Of the 4 total youth sampled,
2 received at least the recommended hours of service. Youth in the sample completed between 4
and 26 hours of service, with an average of 16 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Duval Youth Academy can optimize their SPEP Quality of Service Delivery score by minimizing
gaps in service, by ensuring that internal fidelity monitoring is conducted at least monthly for all
facilitators, and by ensuring that facilitators' performance evaluations include an assessment of
the delivery of IOC
Duval Youth Academy can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Duval Academy - Sequel

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Youth Academy
Sequel Youth Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Living in Balance
SPEP Review Date(s): March 21‐24, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth services Contract Number: 10094
Location: Jacksonville County / Circuit: 4 Number of Beds: 28
Review Date(s): March 21-24, 2017 Lead Reviewer Code: 129
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program’s documentation reflected that the program does have a training protocol and that
three staff received formal training and one of the trained staff has successfully facilitated this
intervention over the past twelve-months. A new therapist has recently been trained and started
conducting group.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a brand named, detailed manual for the delivery of services and provided a
copy for this reviewer to examine. This curriculum does not provide student manuals.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This TAS observed a session of Living in Balance during the review. There were ten youth in
attendance for this intervention. The facilitator provided a copy of lesson from the manual. The
facilitator followed the curriculum plan and stayed true to the fidelity of the product brand. The
facilitator welcomed the youth to participate and stay engaged with the topic.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has recently established a policy of using the Clinical Director to fill in any gaps due
to staff turnover. This service is being provided by the two therapists and during the new hire
process, all new therapists are trained in the Living in Balance curriculum.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Only recently has this become a primary service and there is a policy in place for providing
internal fidelity. This process started during the month of this evaluation. No documentation was
available.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There is a policy in place for corrective action. Program just recently started conducting internal
fidelity monitoring. No correction action has been needed.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
This program has recently switched to a new provider, therefore all evaluations reviewed by this
technical assistance were dated as 90 day evaluations.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 0 youth in the
sample, 0% (0 of 0) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 0 youth in the
sample, 0% (0 of 0) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 0 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 0%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 0%
Moderate to High Score: 0 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 0%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 0 youth
Total Youth in Sample = 0 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 0 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 0% (0 of 0) youth scored Moderate to High-Risk to reoffend, for a score of 0
points.
Of the SPEP sample, 0% (0 of 0) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 0 25 0%
Totals 50 95 53%
This SPEP report evaluates Living in Balance, an intervention delivered at Duval Youth Academy.
LIB was added to the program's list of Primary Services during the week prior to the SPEP review.
The program scored High for Quality of Service Delivery.The program earned 0 points for Amount
of Service: Duration, Amount of Service: Contact Hours, and Risk Level of Youth Served. There
was no data entered into the Juvenile Justice Information system (JJIS) Evidence-Based Services
(EBS) module for any youths. The program must enter dosage data into JJIS in order to receive
any score in this area.
RECOMMENDATION(S):
Duval Youth Academy can optimize their SPEP Amount of Service score by ensuring that dosage
for all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8

service_type: Living in Balance


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Youth Academy
Sequel Youth Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): March 21‐24, 2017
Florida Department of Juvenile Justice
Report Date(s): 8/3/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth services Contract Number: 10094
Location: Jacksonville County / Circuit: 4 Number of Beds: 28
Review Date(s): March 21-24, 2017 Lead Reviewer Code: 129
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 2 # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change (T4C). The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation viewed showed evidence that the program adheres to a formal training
protocol offered through a DJJ approved qualified trainer. The training takes place over a four-day
period requiring 32 hours of skilled instruction. The program has documented two facilitators that
are trained to conduct this intervention with one of them serving in the position of Case Manager
and one serves as the Recreational Therapist. Over the past twelve months, two staff have
actively facilitated this intervention.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator is provided with a physical copy of the Thinking for a Change manual during the
four-day training. There are no individual workbooks for the youth, the manual includes
reproducible handouts and worksheets for the majority of the 25 lessons. The detailed manual is
divided into three sections of concentration: Cognitive Self-Change, Social Skills, and Problem
Solving. The T4C manual is specific to delivery, script, and outline.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program was not able to be observed at the time of review, therefore this indicator rates as
not applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documents and interview with the Clinical Director supported that the program has
had several facilitators leave in the past twelve months. The program has been taken over by a
new provider, therefore, staff turnover has happened. This is the third clinical director in the last
twelve months and there is no clear documentation of intervention accountability before
November 2016. The lack of documentation supports gaps in services. The new clinical director
has a plan for staff turnover in place and is providing direct accountability to facilitators.
The program can earn 2 points by minimizing facilitator turnover and gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program presently has a procedure in place for the last five months to provide internal fidelity
for the facilitator staff. This MQI reviewer viewed documents of three monthly internal fidelity
sheets for this intervention.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the Clinical Director verified that the program has a process for corrective action
in place as needed to support the fidelity monitoring process. After reviewing the documentation
for the six-month period, there were no issues for corrective action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
This MQI reviewer examined the staff evaluations for the trained facilitators of this intervention.
There is no mention of or evaluation of facilitator skills. This was discussed with the clinical
director who will adjust the forms to include facilitator skills for the upcoming evaluation period.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of the facilitators' skill in delivering T4C.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 15 weeks for this type of service. Of the 4 youth in the
sample, 75% (3 of 4) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 4 youth in the
sample, 0% (0 of 4) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 4 youth
Total Youth in Sample = 4 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 100%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 4 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (4 of 4) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 100% (4 of 4) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 76 100 76%
This SPEP report evaluates Impact for Crime (IOC), an intervention delivered at Duval Youth
Academy.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
minimizing facilitator turnover and gaps in service, by ensuring that internal fidelity monitoring is
conducted at least monthly for all facilitators, and by ensuring that facilitators' performance
evaluations include an assessment of the facilitators' skill in delivering T4C.
The program earned 6 points for Amount of Service: Duration. Of the 4 total youth sampled, 3
received at least the recommended weeks of service. Youth in the sample completed between 4
and 22 weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 4 total youth sampled,
none received at least the recommended hours of service. Youth in the sample completed
between 9 and 20 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Duval Youth Academy can optimize their SPEP Quality of Service Delivery score by minimizing
facilitator turnover and gaps in service, by ensuring that internal fidelity monitoring is conducted at
least monthly for all facilitators, and by ensuring that facilitators' performance evaluations include
an assessment of the facilitators' skill in delivering T4C.
Duval Youth Academy can optimize their SPEP Amount of Service score by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Thinking for a Change


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): March 6‐9, 2018
Florida Department of Juvenile Justice
Report Date(s): March 6, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth and Family Services Contract Number: 10094
Location: Duval County / Circuit: 4 Number of Beds: 28
Review Date(s): March 6-9, 2018 Lead Reviewer Code: 141
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
During the review period, eight (8) staff at the facility were eligible to facilitate Impact of Crime
groups, based on a review of training documentation and SkillPro Learning Management System;
3 of the 8 facilitated groups. Documentation indicated each staff was trained by a qualified
trainer. Documentation for this service is maintained in a binder specific to this intervention.
Included in this binder are sign-in sheets, training records, facilitator manual and fidelity
monitoring checklists. All documentation is kept in an orderly, and easy to locate manner. Youth
receive this intervention based on their needs as assessed by the Residential Positive
Achievement Change Tool (R-PACT).
A review of sign-in sheets revealed four cohorts of Impact of Crime were held for the review
period. The first cohort ran 2/28/17 through 6/29/17, beginning with 10 youth and ending with 7
youth. The second cohort ran 7/18/17 through 10/12/17, beginning with 8 youth and ending with
4 youth. The third cohort ran 10/17/17 through 2/1/18, beginning with 11 youth and ending with 9
youth. The fourth cohort began 2/15/18 with 8 youth and was ongoing at the time of the review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The facility utilizes the curriculum Impact of Crime; Addressing the Harm to Victims and the
Community. The program has a manual detailing how Impact of Crime groups are delivered to
youth. The manual gives an overview of the curriculum including number of sessions, and
preparations for groups.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Impact of Crime groups are held on Tuesdays and Thursdays from 1:00 pm to 2:00 pm. The
monitor observed the group conducted on Tuesday, March 8, 2018. The facilitator adhered to the
manual; youth were asked questions based on the curriculum. The group included role playing.
The facilitator utilized the Impact of Crime guide and activity work sheets.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of available documentation indicated there was no facilitator turnover during the review
period. The program has a practice in place to ensure there are no gaps in service due to
facilitator turnover. An interview with the Clinical Director revealed the program is adding multiple
qualified facilitators to avoid any unnecessary gaps in service. If necessary, and as a last resort,
the program may add sessions to each week during a cohort to aid in the completion of the
primary service in the event a staff member submits a two-week notice.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to monitor the delivery of the Impact of Crime intervention.
During the review period, 4 of the 8 qualified staff monitored fidelity for the 3 facilitators. A review
of documentation indicated each facilitator was monitored each month of the review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective action based on fidelity monitoring.
The Clinical Director stated that the program completes fidelity monitorings once a month at the
end of the month. The Clinical Director then follows up with group supervision at the beginning of
the month to discuss a plan of action for any corrective actions that were identified. The Clinical
Director then will readdress any issues during the next month's fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The facilitator's performance evaluation and did include discussion of their skill in delivering the
primary service. The evaluation also indicated the facilitator was selected to facilitate groups
based on education, skills, experience and job description.
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 6 youth in the
sample, 17% (1 of 6) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 6 youth in the
sample, 100% (6 of 6) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 3 youth
Total Youth in Sample = 6 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 3 youth
Total Youth in Sample = 6 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (6 of 6) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 50% (3 of 6) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 75 85 88%
This SPEP report evaluates Impact of Crime, an intervention delivered at Duval Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 6 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 1
received at least the recommended weeks of service. Youth in the sample completed between 12
and 24 weeks of service, with an average of 18 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 6 total youth
sampled, all 10 included dosage in the EBS Module. Of those youth, 6 received at least the
recommended hours of service. Youth in the sample completed between 27 and 53 hours of
service, with an average of 34 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
The failure to include end dates in the EBS Module is a large factor in the reduced Program
Optimization Percentage for this program. It is imperative end dates are entered for all youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Living in Balance
SPEP Review Date(s): March 6‐9, 2018
Florida Department of Juvenile Justice
Report Date(s): March 9, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth and Family Services Contract Number: 10094
Location: Duval County / Circuit: 4 Number of Beds: 28
Review Date(s): March 6-9, 2018 Lead Reviewer Code: 141
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
During the review period, the program had three non-licensed therapists (under the supervision of
a licensed clinician) facilitated Living in Balance groups. At the time of the review, two of those
therapists were still employed at the program. The therapists were trained on the Hazelden
Sessions by the clinical director. The program uses an internal training protocol to ensure training
is consistent among each facilitator.
Documentation for this primary service is maintained in an orderly fashion, in a binder with tabs
clearly indicating each month of activity. Included in this binder are sign in sheets and fidelity
monitoring checklists.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the manual entitled ‘Living in Balance - moving from a life of addiction to a life
of recovery', published by Jeffery Hoffman, Mim J. Landry and Barry Caudill. The program has an
internal training protocol, which includes eight hours of training for each facilitator. Sign in sheets
are used as evidence of this training, and include signatures by the trainee and the Clinical
Director (trainer). Since the last Standardized Program Evaluation Protocol (SPEP) review, the
program has completed 2 cohorts of the primary service, with another 2 in progress at the time of
the review.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The reviewer observed a Living in Balance group on March 6, 2018. The facilitator followed the
lesson plan incorporating substance abuse addiction/relapse and adhered to the manual.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
During the review period, one facilitator left the program. This departure was not unexpected and
occurred between cohorts. There were no gaps in service during the review period. The clinical
director conducts weekly meetings with therapists to prepare/plan prior to assigning facilitators to
groups, in an attempt to avoid any potential gaps in service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director conducts monthly fidelity monitoring. The clinical director completes an
observation checklist which includes a scale to rate the observation of session, facilitator’s self-
evaluation and ideas for improvement, observer's feedback and recommendations, and corrective
action, if needed. A review of documentation indicated the clinical director was consistent in
conducting monthly fidelity monitoring for each facilitor during the review period. Each checklist
included feedback from the facilitator and fidelity monitor, and had signatures from both parties.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective action based on fidelity monitoring.
The Clinical Director stated that the program completes fidelity monitorings once a month at the
end of the month. The Clinical Director then follows up with group supervision at the beginning of
the month to discuss a plan of action for any corrective actions that were identified. The Clinical
Director then will readdress any issues during the next month's fidelity monitoring. The program
did not have any fidelity monitoring noting corrective actions.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The two performance evaluations reviewed included an evaluation of the therapists’ skills in
delivering this primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 3 youth in the
sample, 67% (2 of 3) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 40 contact hours for this type of service. Of the 3 youth in the
sample, 67% (2 of 3) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 1 youth
High = 2 youth
Total Youth in Sample = 3 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 67%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 2 youth
Total Youth in Sample = 3 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (3 of 3) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 67% (2 of 3) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 25 25 100%
Totals 87 95 92%
This SPEP report evaluates Living in Balance, an intervention delivered at Duval Academy.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 3 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 2
received at least the recommended weeks of service. Youth in the sample completed between 31
and 37 weeks of service, with an average of 34 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 3 total youth sampled,
all 3 included dosage in the EBS Module. Of those youth, 2 received at least the recommended
hours of service. Youth in the sample completed between 33 and 99 hours of service, with an
average of 65 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Road
Jacksonville, Florida 32244
Primary Service: Thinking for a Change
SPEP Review Date(s): March 6‐9, 2018
Florida Department of Juvenile Justice
Report Date(s): March 9, 2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Duval Academy MQI Program Code: 1280
Provider Name: Sequel Youth and Family Services Contract Number: 10094
Location: Duval County / Circuit: 4 Number of Beds: 28
Review Date(s): March 6-9, 2018 Lead Reviewer Code: 141
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has three staff members that were actively facilitating Thinking for a Change (T4C)
groups during the review period of March 2017-March 2018. SkillPro was reviewed and two staff
members were trained on 11/5/2017 by a qualified trainer. The third staff member was trained on
3/27/2015 by a qualified trainer. All three staff members completed the required 32 hours of T4C
facilitator training prior to begininng their groups. The staff members who facilitated the groups
were Youth Care Workers. The first cohort began on 7/18/17 and ended on 10/29/17. The first
cohort began with 10 youth and ended with 7 youth. The second cohort began on 12/22/17 and
was in progress during the scheduled review. This cohort began with 11 youth.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has the T4C, Version 3.1 curriculum. The curriculum is detailed with clear
explanation of how each specific session is to be delivered. The manual provides a detailed script
for each of the 25 lessons. The curriculum provides a guide for implementation, the order of the
lessons, and detailed instructions for each specific lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This primary service is provided three times a week (Monday, Wednesday and Friday) by a
certified facilitator. The reviewer observed the Thinking for a Change group on March 7, 2018 and
the facilitator and co-facilitator adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the current documentation revealed that there were no unnecessary gaps in service
during the review period and no facilitator turnover. The Clinical Director stated that she is
training as many of the staff as possible to avoid any future gaps in services. Additionally, the
clinical director conducts weekly meetings with therapists to prepare/plan prior to assigning
facilitators to groups, in an attempt to avoid any potential gaps in service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to monitor the delivery of the Thinking for a Change
intervention. Documentation for four fidelity monitoring sessions were provided for the review
period. Fidelity monitorings were conducted on 8/31/17 and 9/28/17 for one staff, a separate staff
was monitored on 12/27/17, and a third staff was monitored on 2/23/18. All fidelity monitors were
trained by qualified trainers and completed the 32-hour T4C facilitator trianing. An interview with
the Clinical Director revealed the missing documetation was unable to be located.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective action based on fidelity monitoring.
The Clinical Director stated that the program completes fidelity monitorings once a month at the
end of the month. The Clinical Director then follows up with group supervision at the beginning of
the month to discuss a plan of action for any corrective actions that were identified. The Clinical
Director then will readdress any issues during the next month's fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program had three staff members who have facilitated or co-facilitated T4C groups. Two staff
members completed the training in November 2017, and started faciliating groups in December
2017. The third staff member was trained on 3/27/15. Both facilitators have been employed less
than one year. The current group started on December 22, 2017, and scheduled to end on March
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
30, 2018. According to the clinical director, the facilitators performance evaluations will include the
facilitator's skill in delivering the intervention/service. There was no evaluation provided for one
staff who has been working at the facility longer than a year. The Clinical Director was unable to
provide a copy of the evaluation at the time of the re-review. TAS requested that the Clinical
Director email a copy of the evaluation if one was located.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 6 youth in the
sample, 33% (2 of 6) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 6 youth in the
sample, 0% (0 of 6) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 2 youth
High = 4 youth
Total Youth in Sample = 6 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 67%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 6 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (6 of 6) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 67% (4 of 6) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 82 100 82%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Duval Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 6 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 2
received at least the recommended weeks of service. Youth in the sample completed between 18
and 23 weeks of service, with an average of 20 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 6 total youth sampled,
all 6 included dosage in the EBS Module. Of those youth, 0 received at least the recommended
hours of service. Youth in the sample completed between 28 and 39 hours of service, with an
average of 32 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
The failure to include end dates in the EBS Module is a large factor in the reduced Program
Optimization Percentage for this program. It is imperative end dates are entered for all youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10