summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St John's Youth Academy
Sequel TSI of Florida, LLC
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Thinking for a Change
SPEP Review Date(s): April 26‐29, 2016
Florida Department of Juvenile Justice
Report Date(s): 4/17/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St John's Youth Academy QI Program Code: 1266
Provider Name: Sequel TSI of Florida, LLC Contract Number: 10173
Location: St Johns County / Circuit: 7 Number of Beds: 70
Review Date(s): April 26-29, 2016 Lead Reviewer Code: 143
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 1 # Case Managers 5 # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 3 # Personnel Records
Fidelity Monitoring Documents Program Schedules 3 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has three staff (the director of case management and two case managers) trained to
facilitate Thinking for a Change (T4C) groups. The three staff were trained in the most recent
version of T4C by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes Thinking for a Change (T4C), Version 3.1 curriculum. The curriculum is
detailed to the extent it explains how each specific session is delivered and there is a script or
outline for each session.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The monitor observed a T4C group during the review. A copy of the lesson plan was provided for
the reviewer prior to the group. The facilitator followed the material and not skip any portions of
the lesson. The facilitator did not read the material verbatim, but used her own words to express
the topic. The group included youth expressions, role-playing, observations about role-playing,
and feedback from the group. Lastly, the facilitator instructed the group on the homework
assignment for the next session.
d. Facilitator Turnover Basic Score: 2 Point(s)
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was no facilitator turnover and no gaps in service. The most recent T4C group began
September 21, 2015. The program will not schedule a new group until the completion of the
current group.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
This review covered the period from the program opening in July 2015 to the present.
Documentation indicated the program conducted fidelity monitoring of T4C groups. However, the
program did not provide fidelity monitoring reports for September, October, and November 2015.
Several of the fidelity reports were not filled out completely. All fidelity monitoring reviews were
conducted by a facilitator trained in the delivery of the T4C curriculum.
The program can earn 2 points if fidelity monitoring is completed monthly for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Fidelity monitoring reports contained an area for recommendations for the facilitators. The fidelity
monitoring reports reviewed did not identify a need for corrective action related to the delivery of
the primary service. The program provided documentation of the process to use when corrective
actions are necessary.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program has three staff members who are trained to facilitate T4C groups. A review of the
three staff personnel files did not indicate their performance evaluations assessed their skill in the
delivery of the primary service T4C.
The program can earn 1 point if facilitators evaluations include discussion of their skill in delivering
T4C.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 3 youth in the
sample, 33% (1 of 3) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 0 contact hours for this type of service. Of the 3 youth in the
sample, 0% (0 of 3) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 12 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 48%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 3 youth
High = 0 youth
Total Youth in Sample = 3 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 3 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (3 of 3) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 0% (0 of 3) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 12 25 48%
Totals 69 100 69%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St John's Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 3 total youth sampled, only
one youth included an end date, permitting a duration calculation. That youth completed 18 weeks
of service.
The program earned 0 points for Amount of Service: Contact Hours. Of the 3 total youth sampled,
none reached the target hours. Youth completed between 32 and 35 hours.
The program was awarded 12 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St John's Youth Academy can optimize their SPEP Quality of Service Delivery score by
conducting fidelity monitoring monthly and including facilitator skill in delivering T4C in employee
performance evaluations.
St John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Thinking for a Change


state: FL

facility_name: Saint Johns Youth Academy

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth Services
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): March 7‐10, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/15/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St. John's Youth Academy MQI Program Code: 1266
Provider Name: Sequel Youth Services Contract Number: 10173
Location: St. Johns County / Circuit: 7 Number of Beds: 70
Review Date(s): March 7-10, 2017 Lead Reviewer Code: 130
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 3 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 9 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The SPEP review encompasses twelve months of records examinations. The program has just
recently started organizing records for this intervention. It appears two cohorts of Impact of Crime
were conducted during the review period; May to November 2016 and September 2016 to
February 2017. Another cohort was on-going at the time of the review.
The documentation reviewed showed the program has a formal training protocol that is provided
through DJJ from a qualified trainer. There are three staff who received training in this
intervention. One of these three does not conduct group and only provides fidelity monitoring and
coaching.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator has a copy of the manual for Impact of Crime and the reviewer was provided a
copy during the review. There are no separate workbooks for the youth. The manual contains
lessons plans with instructions for implementation and discussion topics.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This intervention is being conducted every Tuesday and Thursday. Youth are chosen based on
their needs as assessed by the RPACT and Youth Needs Assessment Summary. A session of
this intervention was observed during the review. The facilitator was knowledgeable and
comfortable interacting with the youth. The facilitator used Motivational Interviewing skills to
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
create more engagement from the youth. The facilitator was compliant with the fidelity of the
curriculum.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The director of case management was interviewed regarding facilitator turnover and gaps in
services. During the twelve-month review period, there are other names on the attendance sheets
of staff who are no longer with the company, therefore, there have been turnovers. There is clear
evidence of gaps in service found on the attendance sheets. There were weeks and months of no
service due to not prioritizing this intervention. The documentation is very inconsistent and hard to
decipher. New documentation and accountability has been instituted for the last two months.
The program can earn 2 points if there are no gaps in service or facilitator turnover impacting
youth.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was no evidence of internal fidelity monitoring.
The program can earn 2 points by ensuring internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was no evidence of internal fidelity monitoring.
The program can earn 1 point by ensuring the facility has a process by which corrective action is
applied and demonstrated based on fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
There was no evidence of performance evaluations addressing facilitator skill.
The program can earn 1 point by ensuring facilitators' performance evaluations contain an
assessment of their delivery of T4C.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 14 youth in the
sample, 0% (0 of 14) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 14 youth in the
sample, 43% (6 of 14) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 4 youth
High = 7 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 7 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (14 of 14) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 50% (7 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 25 25 100%
Totals 59 85 69%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at St. John's Youth
Academy.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 14 total youth sampled, only
6 included dosage in the EBS Module. Of those youth with dosage, none included an end date,
rendering it impossible to calculate an accurate end date for the sampled youth.
The program earned 4 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, only 6 included dosage in the EBS Module. Of those youth with dosage, all received at
least the recommended hours of service. Youth in the sample completed 24 hours of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can optimize their SPEP Quality of Service Delivery score by
minimizing facilitator turnover and gaps in service, by ensuring internal fidelity monitoring is
conducted at least monthly for all facilitators, by ensuring the facility has a process by which
corrective action is applied and demonstrated based on fidelity monitoring, and by ensuring
facilitators' performance evaluations contain an assessment of their delivery of IOC.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
is recorded accurately in the EBS Module and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Saint Johns Youth Academy

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth Services
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): March 7‐10, 2017
Florida Department of Juvenile Justice
Report Date(s): 1/15/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St. John's Youth Academy MQI Program Code: 1266
Provider Name: Sequel Youth Services Contract Number: 10173
Location: St. Johns County / Circuit: 7 Number of Beds: 70
Review Date(s): March 7-10, 2017 Lead Reviewer Code: 130
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor 3 # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change (T4C). The program was awarded
30 points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 5 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 25%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 3 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The SPEP review encompasses twelve months of records examinations. The program conducted
one cycle of T4C during the review period; January through June 2016. Another cohort was on-
going at the time of the review. This intervention is held on Mondays and Wednesdays. Youth are
assigned to this intervention based on the RPACT and the Youth Needs Assessment Summary.
Thinking for a Change is a closed group curriculum. The program maintains records by keeping
all documentation in a large binder for evidence based groups.
The documentation reviewed showed the program adheres to a formal training protocol offered
through a DJJ approved qualified trainer. The training takes place over a four-day period requiring
32 hours of skilled instruction. The program has documentation for five facilitators in this
intervention. Over the past twelve months, two staff have actively facilitated this intervention.
Program utilizes the co-facilitation style of conducting this intervention.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Each facilitator is provided with a physical copy of the Thinking for a Change manual during the
four-day training. There are no individual workbooks for the youth, the manual includes
reproducible handouts and worksheets for the majority of the 25 lessons. The detailed manual is
divided into three sections of concentration: Cognitive Self-Change, Social Skills, and Problem
Solving. The T4C manual is specific as to delivery.
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A session of Thinking for a Change was observed during the review. The lesson title was
Introduction. There were ten youth in attendance for this group. The facilitators provided a copy of
lesson from the manual for the observer to follow along. The facilitators had some issues with
following the lesson according to the manual. No visual aids were prepared or used for this
lesson which is inconsistent with the curriculum.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documents and interview with the Director of Case Management identified the
program has had facilitator turnovers in the past twelve months: three facilitators left the agency.
This turnover and a facilitator’s medical emergency caused a gap in service. In the event of future
turnovers, the program will have several trained facilitators that are prepared to rotate into position
until other staff are trained to replace the missing facilitators. During this review period, there are
over six months with no evidence of this intervention being conducted: from June 2016 through
February 2017.
The program can earn 2 points if there are no gaps in service or facilitator turnover impacting
youth.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was no evidence of internal fidelity monitoring.
The program can earn 2 points by ensuring internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was no evidence of internal fidelity monitoring.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 1 point by ensuring the facility has a process by which corrective action is
applied and demonstrated based on fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
There was no evidence of performance evaluations addressing facilitator skill.
The program can earn 1 point by ensuring facilitators' performance evaluations contain an
assessment of their delivery of T4C.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 0 youth in the
sample, 0% (0 of 0) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 0 youth in the
sample, 0% (0 of 0) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 0 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 0%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 0%
Moderate to High Score: 0 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 0%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 0 youth
Total Youth in Sample = 0 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 0 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 0% (0 of 0) youth scored Moderate to High-Risk to reoffend, for a score of 0
points.
Of the SPEP sample, 0% (0 of 0) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 5 20 25%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 0 25 0%
Totals 40 100 40%
This SPEP report evaluates Thinking for a Change (T4C), an intervention delivered at St. John's
Youth Academy.
The program scored Low for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration and Amount of Service: Contact
Hours. There was no T4C data entered into the Juvenile Justice Information System (JJIS)
Evidence-Based Services Module (EBS) Module for youth who completed the program during the
time period reviewed. The program must enter dosage data into JJIS in order to receive any score
in this area. Without data in the EBS Module, a T4C sample could not be constructed.
The program was awarded 0 available points for Risk Level of Youth Served. Without data in the
EBS Module, a T4C sample could not be constructed. This is calculated using data from the
Community - Positive Achievement Change Tool (C-PACT) assessment. This score reflects
youths' most recent C-PACT score prior to placement at the program. The program itself has no
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
control over youths' C-PACT risk level because the scored assessment was administered just
prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can optimize their SPEP Quality of Service Delivery score by adhering
to the manual during groups, by minimizing facilitator turnover and gaps in service, by ensuring
internal fidelity monitoring is conducted at least monthly for all facilitators, by ensuring the facility
has a process by which corrective action is applied and demonstrated based on fidelity
monitoring, and by ensuring facilitators' performance evaluations contain an assessment of their
delivery of T4C.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage data is included in the EBS Module and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): February 8‐9, 2018
Florida Department of Juvenile Justice
Report Date(s): 2/9/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St Johns Youth Academy MQI Program Code: 1266
Provider Name: Sequel Youth and Family Services Contract Number: 10173
Location: St Augustine County / Circuit: 7 Number of Beds: 70
Review Date(s): 2/8/18 Lead Reviewer Code: XXXX
Persons Interviewed
Program Director Corporate QI/QA staff 2 # Program Supervisors
DJJ Monitor 3 # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 9 # Youth Records (Closed)
Staff Evaluations Table of Organization 3 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of SkillPro and interview with the case management supervisor and assistant facility
administrator (AFA) revealed that all facilitators have received the Impact of Crime (IOC) facilitator
training for the required twenty-eight hours. Fidelity checks are completed monthly of each cohort.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Review of the training materials reveal that the program uses the Impact of Crime (IOC)
manualized curriculum for instructor presentation and youth assignment/activities. Youth were
observed completing assignments out of the approved IOC manual.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Observation of the Impact of Crime (IOC) group at 6 pm on February 8, 2018, revealed the
curriculum is being implemented with fidelity at the facility. Both instructors were utilizing the
approved materials and youth were completing assignments/activities out of the book.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Review of services over the past year have shown there has been some change in facilitators
who implement the group and that in some cases, entire cohorts were aborted and in others, a
significant percentage of youth did not complete the curriculum. Recent cohorts have a higher
completion rate and facilitators have stabilized.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation provided illustrates the monthly monitoring of each cohort has taken place. These
monitoring activities are done by the Director of Case Management, who is also a trained Impact
of Crime (IOC) facilitator.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Adjustments to the facilitators who implement the criteria and the report of administrative staff,
demonstrate that corrective actions based on monitoring have resulted in a change of instructors
and improved completion rate.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Evaluations show that staff are assessed on their ability to deliver the curriculum with fidelity.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 9 youth in the
sample, 22% (2 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 9 youth in the
sample, 78% (7 of 9) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 78%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 2 youth
High = 5 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 56%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 78% (7 of 9) youth scored Moderate to High-Risk to reoffend, for a score of
12 points.
Of the SPEP sample, 56% (5 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 20 25 80%
Totals 68 85 80%
This SPEP report evaluates Impact of Crime, an intervention delivered at St. John's Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 9 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those 2 youth with correct dosage, both
received at least the recommended weeks of service. Youth in the sample completed 16 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 9 total youth sampled,
all 9 included dosage in the EBS Module. Of those youth, 7 received at least the recommended
hours of service. Youth in the sample completed between 7 and 36 hours of service, with an
average of 26 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of the review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced Program
Optimization Percentage for this program. It is imperative end dates are entered for all youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Thinking for a Change (T4C)
SPEP Review Date(s): February 8‐9, 2018
Florida Department of Juvenile Justice
Report Date(s): 2/9/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St Johns Youth Academy MQI Program Code: 1266
Provider Name: Sequel Youth and Family Services Contract Number: 10173
Location: St Augustine County / Circuit: 7 Number of Beds: 70
Review Date(s): 2/8/18 Lead Reviewer Code: XXXX
Persons Interviewed
Program Director Corporate QI/QA staff 2 # Program Supervisors
DJJ Monitor 3 # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 9 # Youth Records (Closed)
Staff Evaluations Table of Organization 3 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has two staff certified to facilitate the Thinking for a Change (T4C) groups. The staff
were trained in the most recent version of T4C by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes Thinking for a Change (T4C) manual and curriculum. The T4C curriculum is
detailed to the extent it explains how each specific session is delivered and a script to use, which
outlines each session.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Thinking for a Change (T4C) group was observed during the review. A copy of the lesson
plan was provided for review. The facilitators followed the T4C material. The group included role-
playing, observations from youth on role-play, utilizing youth to write points of the lesson,
homework, and continious feedback from the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program reported no facilitator turnover and no gaps in service. The most recent Thinking for
a Change (T4C) group began in January 2018. The program will schedule a new T4C group at
the completion of the current group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program reported monthly fidelity monitoring of the Thinking for a Change (T4C) groups.
Fidelity monitoring is completed by the Assistant Facility Administrator for Operations. Six months
of fidelity monitoring forms were reviewed and confirmed monthly monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Fidelity monitoring reports contained an area for recommendations from the fidelity monitor. The
fidelity monitoring reports observed did not identify a need for corrective action and the monitor
did not report any corrective action needed for facilitators. The program reported possible
corrective action steps taken to needed facilitators.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Evaluations show that staff are assessed on their ability to deliver the curriculum with fidelity.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 9 youth in the
sample, 22% (2 of 9) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 45 contact hours for this type of service. Of the 9 youth in the
sample, 100% (9 of 9) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 4 youth
High = 4 youth
Total Youth in Sample = 9 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 44%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 9 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (8 of 9) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 44% (4 of 9) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 90 100 90%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St. John's Youth
Academy.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 9 total youth sampled, only
2 included dosage with end dates in the EBS Module. Of those 2 youth with correct dosage, both
received at least the recommended weeks of service. Youth in the sample completed between 21
and 22 weeks of service, with an average of 21 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 9 total youth
sampled, all 9 included dosage in the EBS Module. Of those youth with correct dosage, 9
received at least the recommended hours of service. Youth in the sample completed between 48
and 58 hours of service, with an average of 49 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
The failure to include end dates in the EBS Module is a large factor in the reduced Program
Optimization Percentage for this program. It is imperative end dates are entered for all youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St Augustine, Florida 32085
Primary Service: Skillstreaming the Adolescent
SPEP Review Date(s): February 8-9, 2018
Florida Department of Juvenile Justice
Report Date(s): 2/9/18
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: St Johns Youth Academy MQI Program Code: 1266
Provider Name: Sequel Youth and Family Services Contract Number: 10173
Location: St Augustine County / Circuit: 7 Number of Beds: 70
Review Date(s): 2/8/18 Lead Reviewer Code: XXXX
Persons Interviewed
Program Director Corporate QI/QA staff 2 # Program Supervisors
DJJ Monitor 3 # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 4 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 9 # Youth Records (Closed)
Staff Evaluations Table of Organization 3 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent. The program was
awarded 15 points because the Primary Service is identified as a Group 3 Service. The specific
Sub-Component Service Type identified is Social Skills Training. The Primary Service was
identified as this type of service as it focuses on developing social skills required for an individual
to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has staff certified to facilitate the Skillstreaming the Adolescent groups. The staff
were trained in the most recent version of Skillstreaming the Adolescent by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Skillstreaming the Adolescent manual and curriculum. The manual
includes the order of lessons to be presented and contains lesson plans with instructions for
providing each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Skillstreaming the Adolescent group was observed during the review. The facilitators followed
the Skillstreaming lesson plan material. The group included role-playing, observations from youth
on role-play, homework, and continuous feedback from the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program reported no facilitator turnover and no gaps in service. There are multiple trained
facilitators at this program. The program attempts to provide a trained facilitator for each youth
dorm. This allows for coverage of a group and limits interruption of service delivery.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program reported monthly fidelity monitoring of Skillstreaming groups. Fidelity monitoring is
completed by the Assistant Facility Administrator for Operations. Six months of fidelity monitoring
forms were reviewed and confirmed monthly monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Fidelity monitoring reports contained an area for recommendations from the fidelity monitor. The
fidelity monitoring reports observed did not identify a need for corrective action and the monitor
did not report any corrective action needed for facilitators. The program reported possible
corrective action steps taken to needed facilitators.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Evaluations show that staff are assessed on their ability to deliver the curriculum with fidelity.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 7 youth in the
sample, 0% (0 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 7 youth in the
sample, 71% (5 of 7) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 86%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 3 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 43%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 3 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 86% (6 of 7) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 43% (3 of 7) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 69 85 81%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at St. John's
Youth Academy.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 7 total youth sampled, only
5 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 8
and 12 weeks of service, with an average of 9 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 7 total youth sampled,
all 7 included dosage in the EBS Module. Of those youth with correct dosage, 5 received at least
the recommended hours of service. Youth in the sample completed between 1 and 36 hours of
service, with an average of 26 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered just prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Skillstreaming the Adolescent