state: FL

facility_name: Alachua Academy

summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel TSI of Florida, LLC
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): January 10‐13, 2017
Florida Department of Juvenile Justice
Report Date(s): 6/12/17
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Alachua Academy MQI Program Code: 1165
Provider Name: Sequel TSI of Florida, LLC Contract Number: 10189
Location: Alachua County / Circuit: 8 Number of Beds: 24
Review Date(s): January 10-13, 2107 Lead Reviewer Code: 130
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff 1 # Other (listed by title): IOC
DMHCA or designee # Healthcare Staff Facilitator
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 1 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact Of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 5 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 25%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 3 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has identified one facilitator for Impact of Crime. Their training files document that
she has been trained by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the Department’s Impact of Crime (IOC) curriculum. The program utilizes
the IOC facilitator and student guides. There are five objectives with twenty-four lessons in total.
The curriculum includes implementation guidelines, identifies the order the lessons are to be
delivered, and detailed instructions for each lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime (IOC) group was not able to be observed during the review week because the
IOC session was conducted not as written on schedule.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
There was a gap in the delivery of Impact of Crime (IOC) and high facilitator turnover since the
new contract effective date in February, 2016. The last session of IOC was started in October,
2016.
The program can earn 2 points if IOC is delivered consistently with no gaps in service of one
month or more and if facilitator turnover is minimized.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 0Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has not conducted fidelity monitoring for this primary service by trained facilitators.
The program can earn 2 points if fidelity monitoring is completed at least monthly for each
facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 0Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program does not have a process for corrective action based on fidelity monitoring. No
corrective action was required because no fidelity monitoring was completed.
The program can earn 1 point by developing and implementing a plan for corrective action based
on fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program has an employee performance appraisal form that did not include an evaluation of
the facilitator’s skill in delivering a primary service. The program created a new evaluation form
that will be used and includes an area for performance delivering an intervention.
The program can earn 1 point if each facilitator is evaluated on their skill in delivery of IOC.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 2 youth in the
sample, 0% (0 of 2) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 2 youth in the
sample, 0% (0 of 2) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 0 youth
High = 2 youth
Total Youth in Sample = 2 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 100%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 2 youth
Total Youth in Sample = 2 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (2 of 2) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 100% (2 of 2) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 5 20 25%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 200%
Totals 50 85 59%
This SPEP report evaluates Impact Of Crime (IOC), an intervention delivered at Alachua
Academy.
The program scored 5 for Quality of Service Delivery. This score can be optimized if the
intervention is delivered, if the program minimizes facilitator turnover and gaps in service, if the
program establishes a process for internal fidelity monitoring and corrective action based on
fidelity monitoring, and conducts internal fidelity monitoring at least monthly for each facilitator.
The program earned 0 points for Amount of Service: Duration. Of the 2 total youth sampled, both
included dosage with end dates in the EBS Module. Neither received at least the recommended
weeks of service. Both youth completed exactly 0.29 weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 2 total youth sampled,
both included dosage in the EBS Module. Neither received at least the recommended hours of
service. Both youth completed exactly 1 hour of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score if the intervention is
delivered, if the program minimizes facilitator turnover and gaps in service, if the program
establishes a process for internal fidelity monitoring and corrective action based on fidelity
monitoring, and conducts internal fidelity monitoring at least monthly for each facilitator.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Alachua Academy - Sequel

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel TSI of Florida, LLC
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Thinking For A Change (T4C)
SPEP Review Date(s): January 10‐13, 2017
Florida Department of Juvenile Justice
Report Date(s): 6/12/17
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Alachua Academy MQI Program Code: 1165
Provider Name: Sequel TSI of Florida, LLC Contract Number: 10189
Location: Alachua County / Circuit: 8 Number of Beds: 24
Review Date(s): January 10-13, 2107 Lead Reviewer Code: 37
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff 1 # Other (listed by title): Assistant
DMHCA or designee # Healthcare Staff Program Director
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking For A Change (T4C). The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 5 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 25%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 2 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The assistant facility administrator was previously trained in Thinking for a Change (T4C), but has
not received the updated training for the most current version of T4C, which is the T4C Version
3.1. The program only has one staff trained in the most recent version of T4C. This staff has not
facilitated the course, and can only facilitate the first course as a co-facilitator. With no other staff
trained in the most recent version of T4C, the program has not provided the course.
The program can earn 1 point if staff get trained in the most recent version of T4C.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has the T4C, Version 3.1 curriculum. The curriculum is detailed to the extent it
explains how each specific session is delivered and there is a script or outline for each session.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
There were no T4C groups conducted during the annual compliance review; therefore, this
indicator rates as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program reported they last conducted a T4C group in September, 2016, but did not have
documentation of any groups. The program reported they have not provided T4C groups due to
two staff who had been trained in T4C leaving their employment at the program. The program
cannot provide T4C groups until more staff are trained as facilitators.
The program can earn 2 points if facilitators conduct T4C groups with consistent facilitators and
minimal gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 0Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program did not have any documentation of T4C groups conducted during the review period,
thus there was no documentation of fidelity monitoring being conducted.
The program can earn 2 points if fidelity monitoring of T4C groups are conducted and internal
fidelity monitoring is completed at least monthly for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 0Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was no documentation of T4C groups conducted by the program during the review period,
thus no fidelity monitoring was conducted and no corrective actions based on fidelity monitoring
was taken.
The program can earn 1 point if fidelity monitoring of T4C groups are conducted.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program has been in operation under the current contract for less than a year, thus annual
performance evaluations have not yet been completed. The program updated their performance
evaluation forms during the review to include the delivery of specialized services, but no staff
have yet been evaluated on the delivery of T4C.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 3 youth in the
sample, 0% (0 of 3) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 3 youth in the
sample, 0% (0 of 3) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 67.%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 0 youth
Moderate-High = 2 youth
High = 0 youth
Total Youth in Sample = 3 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 3 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 67% (2 of 3) youth scored Moderate to High-Risk to reoffend, for a score of
5 points.
Of the SPEP sample, 0% (0 of 3) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 5 20 25%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 5 25 20%
Totals 45 100 45%
This SPEP report evaluates Thinking For A Change (T4C), an intervention delivered at Alachua
Academy.
The program scored Low for Quality of Service Delivery. This score can be optimized by ensuring
that facilitators are trained to deliver T4C, by delivering T4C, by minimizing facilitator turnover and
gaps in service, by conducting internal fidelity monitoring monthly for each facilitator, and by
establishing and following a corrective action plan based on fidelity monitoring.
The program earned 0 points for Amount of Service: Duration. Of the 3 total youth sampled, none
included dosage with end dates in the EBS Module. Without dosage in the EBS Module, it is
impossible to calculate amount of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 3 total youth sampled,
none included dosage in the EBS Module. Without dosage in the EBS Module, it is impossible to
calculate amount of service.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring that
facilitators are trained to deliver T4C, by delivering T4C, by minimizing facilitator turnover and
gaps in service, by conducting internal fidelity monitoring monthly for each facilitator, and by
establishing and following a corrective action plan based on fidelity monitoring.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Thinking for a Change


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel TSI of Florida, LLC
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): January 24, 2018
Florida Department of Juvenile Justice
Report Date(s): 05/23/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Alachua Academy MQI Program Code: 1165
Provider Name: Sequel TSI of Florida, LLC Contract Number: 10189
Location: Alachua County / Circuit: 8 Number of Beds: 28
Review Date(s): January 24, 2018 Lead Reviewer Code: 37
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 2 # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group sign-in sheets
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has had two facilitators provide Impact of Crime groups during the review period.
Their training files document each has been trained by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the Department’s Impact of Crime (IOC) curriculum. The program utilizes
the IOC facilitator and student guides. The curriculum includes implementation guidelines,
identifies the order the lessons are to be delivered, and provides detailed instructions for each
lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime (IOC) group was not able to be observed during the review because the last
IOC session was conducted in November 2017.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program completed two cohorts of Impact of Crime (IOC) during the review period, with the
last cohort being completed on November 25, 2017. The program is scheduled to begin another
cohort this week, as there are now enough youth needing the course. Each cohort was provided
by a trained facilitator without any gap of service due to staff turnover. Two staff have facilitated
IOC groups during the review period and both are still employed at the program.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Two cohorts of Impact of Crime (IOC) groups were provided, with one provided from March 14 to
July 22, 2017, and the other provided from August 29 to November 25, 2017. Based on the dates
the cohorts were provided and the requirement for monthly fidelity monitoring, there should have
been seven instances of fidelity monitoring documented. Fidelity monitoring was documented
three times during the review period, which occurred on May 30, June 28, and October 27, 2017.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective action based on fidelity monitoring. The program has a
checklist to be completed for fidelity monitoring, which includes an evaluation of the session and
any corrective action required based on that observation. Fidelity monitoring completed did not
identify the need for corrective action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Neither facilitator of Impact of Crime (IOC) had a performance evaluation completed in the past
year; therefore, this indicator rates as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 7 youth in the
sample, 71% (5 of 7) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 7 youth in the
sample, 100% (7 of 7) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 12%
Table 1
Moderate = 1 youth
Moderate-High = 0 youth
High = 6 youth
Total Youth in Sample = 7 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 86%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 6 youth
Total Youth in Sample = 7 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (7 of 7) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 86% (6 of 7) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime, an intervention delivered at Alachua Academy.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring fidelity monitoring is completed every month for each facilitator and by ensuring each
facilitator's performance evaluation includes discussion of their skill in delivering Impact of Crime.
The program earned 6 points for Amount of Service: Duration. Of the 7 total youth sampled, all 7
included dosage with end dates in the EBS Module. Of those youth, 5 received at least the
recommended weeks of service. Youth in the sample completed between 12 and 18 weeks of
service, with an average of 17 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 7 total youth
sampled, all 7 included dosage in the EBS Module. Of those youth, 7 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 24 and 27 hours of
service, with an average of 26 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring fidelity
monitoring is completed every month for each facilitator and by ensuring each facilitator's
performance evaluation includes discussion of their skill in delivering Impact of Crime.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel Youth and Family Services
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Living in Balance
SPEP Review Date(s): January 24, 2018
Florida Department of Juvenile Justice
Report Date(s): 05/23/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Alachua Academy MQI Program Code: 1165
Provider Name: Sequel Youth and Family Services Contract Number: 10189
Location: Alachua County / Circuit: 8 Number of Beds: 28
Review Date(s): January 24, 2018 Lead Reviewer Code: 37
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks 2 # Personnel Records
Fidelity Monitoring Documents Program Schedules 2 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group sign-in sheets
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living in Balance. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program has had four facilitators provide Living in Balance groups during the review period.
Their training files documented each has been trained by a qualified trainer. Two of four
facilitators are no longer with the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the Living in Balance curriculum and utilizes the Living in Balance Facilitator
Guide. The facilitator guide includes implementation guidelines, identifies the order the lessons
are to be delivered, and detailed instructions for each lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The primary service, Living in Balance, was not able to be observed during the review.
At the time of the review, there was a concern of a youth's reaction to an outsider observing
groups. The youth was prone to behavioral issues when visitors were on the wing, and it was
deemed in the best interest of the remaining youth in the group to forego an observation. This
reduced point will not be counted against the program, as all parties involved in this review were
being responsive to a youth's trauma history.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Group sign-in sheets reviewed found Living in Balance groups were provided during the review
period without gaps in service. The sign-in sheets showed, however, that facilitators were used
interchangeably with different groups of youth and two facilitators who were providing Living in
Balance groups are no longer with the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation of fidelity monitoring for Living in Balance found fidelity monitoring was completed
from April through June 2017 and from October through December 2017. These did not include
monitoring for each facilitator each month consistently. There was not any fidelity monitoring
documented from July through September 2017.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective action based on fidelity monitoring. The program has a
checklist to be completed for fidelity monitoring, which includes an evaluation of the session and
any corrective action required based on that observation. Fidelity monitoring completed did not
identify the need for corrective action.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the four personnel files for staff who facilitated Living in Balance groups found
performance evaluations were completed for two of the staff. These evaluations were completed
in March 2017 and included the delivery of Living in Balance groups. These two staff are no
longer with the program. The remaining two staff who facilitate Living in Balance groups did not
have a performance evaluation completed as of the time of the review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 14 youth in the
sample, 93% (13 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 14 youth in the
sample, 14% (2 of 14) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 3 youth
High = 9 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 64%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (13 of 14) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 64% (9 of 14) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 81 95 85%
This SPEP report evaluates Living in Balance, an intervention delivered at Alachua Academy.
The program scored High for Quality of Service Delivery. This score can be optimized by
ensuring fidelity monitoring is completed every month for each facilitator, and ensuring facilitator
turnover does not impact groups.
While the sum score for this program was only 7 out of 10 available points, the overall quality was
not deemed appropriate to be reduced. The reduced point for Observed Adherence to the
Manual/Protocol will not be counted against the program, as all parties involved in this review
were being responsive to a youth's trauma history when the decision was made to forego the
observation.
The program earned 8 points for Amount of Service: Duration. Of the 14 total youth sampled, all
14 included dosage with end dates in the EBS Module. Of those youth, 13 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended weeks of service. Youth in the sample completed between 15 and 55 weeks of
service, with an average of 35 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 2 received at least the
recommended hours of service. Youth in the sample completed between 23 and 45 hours of
service, with an average of 35 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring fidelity
monitoring is completed every month for each facilitator, and ensuring facilitator turnover does not
impact groups.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Living in Balance


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel Youth and Family Services
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: VOICES
SPEP Review Date(s): January 24, 2018
Florida Department of Juvenile Justice
Report Date(s): 05/23/2018
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Alachua Academy MQI Program Code: 1165
Provider Name: Sequel Youth and Family Services Contract Number: 10189
Location: Alachua County / Circuit: 8 Number of Beds: 28
Review Date(s): January 24, 2018 Lead Reviewer Code: 37
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers 5 # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group sign-in sheets
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is VOICES. The program was awarded 25 points because
the Primary Service is identified as a Group 4 Service. The specific Sub-Component Service
Type identified is Group Counseling. The Primary Service was identified as this type of service as
it focuses on psychological or interpersonal problems or issues faced by an individual and
involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had five staff trained to facilitate VOICES groups in March 2017. Documentation
showed, however, that VOICES groups began in February 2017. Three of the staff trained to
facilitate VOICES groups are no longer with the program.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the VOICES curriculum and utilizes the VOICES Facilitator Guide. The
facilitator guide includes implementation guidelines, identifies the order the lessons are to be
delivered, and detailed instructions for each lesson.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The primary service, VOICES, was not able to be observed during the review.
At the time of the review, there was a concern of a youth's reaction to an outsider observing
groups. The youth was prone to behavioral issues when visitors were on the wing, and it was
deemed in the best interest of the remaining youth in the group to forego an observation. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
reduced point will not be counted against the program, as all parties involved in this review were
being responsive to a youth's trauma history.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Group sign-in sheets reviewed found three cohorts of the primary service, VOICES, were
completed during the review period. A fourth cohort was in progress at the time of the review. One
completed cohort was provided by one facilitator without a gap in service. Another completed
cohort had a change in the facilitator without a gap in service. The most recent completed cohort
had three different facilitators, with the facilitator beginning the cohort going out on extended sick
leave. The cohort currently being delivered was started by one facilitator and another facilitator
joined as a co-facilitator after multiple sessions had been provided. Three facilitators who were
providing VOICES groups are no longer with the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation of fidelity monitoring for VOICES found fidelity monitoring was provided monthly
for the current cohort from October through December 2017, though not for each facilitator. There
was no fidelity monitoring completed for any sessions of the three completed cohorts of VOICES
provided during the review period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process for corrective action based on fidelity monitoring. The program has a
checklist to be completed for fidelity monitoring, which includes an evaluation of the session and
any corrective action required based on that observation. Fidelity monitoring completed did not
identify the need for corrective action, though fidelity monitoring was not completed for three
cohorts of VOICES and was not completed for each facilitator monthly for the current cohort of
VOICES.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the four applicable personnel files for staff who facilitated VOICES groups found
performance evaluations were completed for three of the staff. These evaluations included the
delivery of VOICES groups. Two of these staff are no longer with the program. The remaining
staff who facilitated VOICES groups during the review period did not have a performance
evaluation completed yet.
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 13 youth in the
sample, 0% (0 of 13) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 13 youth in the
sample, 0% (0 of 13) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 2 youth
High = 9 youth
Total Youth in Sample = 13 youth
Percentage of Youth with High-Risk Level to Reoffend: 69%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (12 of 13) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 69% (9 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 63 95 66%
This SPEP report evaluates VOICES, an intervention delivered at Alachua Academy.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring staff are trained to facilitate VOICES groups prior to delivering services, and ensuring
facilitator turnover does not impact groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Duration. Of the 13 total youth sampled, all
13 included dosage with end dates in the EBS Module. Of those youth, 0 received at least the
recommended weeks of service. Youth in the sample completed between 11 and 18 weeks of
service, with an average of 14 weeks.
The program earned 0 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed between 12 and 27 hours of
service, with an average of 18 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring staff are
trained to facilitate VOICES groups prior to delivering services, and ensuring facilitator turnover
does not impact groups.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: VOICES: A Program of Self-Exploration and Empowerment


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Sequel Youth and Family Services
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Impact of Crime
SPEP Review Date(s): 12/11-14/2018
Florida Department of Juvenile Justice
Report Date(s): 1/2/2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7* Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes the Impact of Crime curriculum as a primary service. A review of the
documentation from the prior 12 months showed there were eight staff members trained in the
primary service. Two of the trained staff are no longer employed with the facility. The six trained
staff are two case managers, the Clinical Director, the Executive Director, the former Recreation
Therapist, and a shift supervisor. Both case managers were trained 6/26/18-6/29/18. The Clinical
Director and Executive Director were trained on 2/20/18-2/23/18. The former Recreation
Therapist was trained on 4/14/17 and the shift supervisor was trained on 9/2/16. The four-day
training was provided by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm v 2.0 as the manual. The
facilitator's manual is divided into 7 chapters, an appendix with handouts, and the student guide
section (which includes the student exercises). The manual addresses the order in which to
deliver the content and gives the facilitators an outline to follow. The facility does not have
separate student workbooks to give out. The facilitators make copies of the exercises before
each session to pass out to the youth. The group is a closed group with no more than 12 youth in
each cohort. The curriculum is to be delivered to the youth over a minimum of 16 weeks for 32
sessions.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The final session of the current cohort was held outside of the time the reviewer was on-site.TAS
was unable to observe an Impact of Crime session. This curriculum was not able to be observed;
therefore, this indicator rates as non-applicable.
*This observation did not occur because the intervention concluded as scheduled, in adherence
with the curriculum design, prior to an observation taking place; therefore, this unavailable point
will not be counted against the program's overall score.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
According to the documentation reviewed, there has been facilitator turnover at the program.
There were eight staff members who were trained to facilitate Impact of Crime; two of the eight
are no longer employed with the program. The program documentation from the prior 12 months
shows there were four facilitators running groups during the review period. There was one gap in
services greater than one month from February to April 2018. One cohort began in January 2018
and ended before completing the curriculum in February 2018. Another cohort began 4/4/18 and
ended 7/8/18. Two additional cohorts began August 6th and 7th, 2018, and ran until December
2018. According to the Clinical Director, the program tries to have at least three staff trained to
facilitate the Impact of Crime group in the event of staff turnover.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the review period indicated fidelity monitoring was conducted
during eight of the 12 months. Documentation of fidelity monitoring was not available for January
2018 through April 2018. Documentation confirmed all monitoring was conducted by qualified
staff trained in the curriculum; one was completed by the Executive Director, the rest were
completed by the Clinical Director.
The program has a process in place for internal fidelity monitoring. The Clinical Director conducts
monthly fidelity monitoring of each facilitator. The fidelity monitoring is discussed with the
facilitator to ensure the facilitator is completing groups to the fidelity of the model. According to the
Clinical Director, if she is unavailable to complete the fidelity monitoring, she will allow another
staff member to conduct the fidelity monitoring if that staff member is trained to facilitate IOC.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on the results of fidelity
monitoring. The program's corrective action process begins with the monthly internal fidelity
monitoring completed by the Clinical Director. The Clinical Director conducts the fidelity
monitoring and then discusses any findings with the facilitator. The Clinical Director will give the
facilitator two weeks to make corrections and then the Clinical Director will re-evaluate the
facilitator. According to the reviewed documentation for the prior 12 months, there was no
corrective action noted on the fidelity monitoring reports.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Evaluations were reviewed for the three facilitators that were completed during the review period.
Each evaluation has an area on the evaluation called "Facilitator Skill Delivering the Evidence-
Based Programming". Each of the evaluations reviewed had specific information based on the
facilitator's delivery of the evidence-based programming.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 25 youth in the
sample, 36% (9 of 25) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 25 youth in the
sample, 64% (16 of 25) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 96%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 7 youth
Moderate-High = 5 youth
High = 12 youth
Total Youth in Sample = 25 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 12 youth
Total Youth in Sample = 25 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 96% (24 of 25) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 48% (12 of 25) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 25 25 100%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Alachua Academy.
26 youth completed the program during the review period. Of those youth, 25 had not previously
received this intervention in another setting.
The program scored High for Quality of Service Delivery.
While the sum score for this program was only 7 out of 10 available points, it was not deemed
appropriate to reduce the overall quality. The reduced point for Observed Adherence to the
Manual/Protocol will not be counted against the program as the observation was unable to be
conducted because this intervention concluded as scheduled, in adherence with the curriculum
design, prior to an observation taking place.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Duration. Of the 25 total youth sampled, only
20 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 9
received at least the recommended weeks of service. Youth in the sample completed between 3
and 24 weeks of service, with an average of 16 weeks.
The program earned 6 points for Amount of Service: Contact Hours. Of the 25 total youth
sampled, only 20 included dosage in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended hours of service. Youth in the sample completed between 2
and 37 hours of service, with an average of 25 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring there
are no unnecessary gaps in service, and by ensuring fidelity monitoring is completed every month
for each facilitator.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9