summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Wayne Halfway House
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Living In Balance (LIB)
SPEP Review Period: February 1, 2023-January 31, 2024
SPEP Review Conducted: 3/13/2024
Florida Department of Juvenile Justice
Report Date: 7/19/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Living In Balance.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the program
used Living in Balance (LIB) as a primary service. Six (6) staff were trained in the primary service.
The six (6) trained staff were the clinical director and five (5) therapists. The training is a two-hour
in-service training delivered by the clinical director. The program has a written protocol for the in-
service training which documents the schedule. The documentation showed all six (6) trained staff
conducted groups. The clinical director, corporate clinical director, and a trained therapist
conducted the monthly internal fidelity monitoring. A trained staff member can conduct the
monthly fidelity monitoring. The primary service is held on Mondays and Wednesdays.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Documentation for the annual review period of 2/1/2023 through January 31, 2024 shows the
program utilizes the Living in Balance: Moving from a Life of Addiction to a Life of Recovery as the
manual. Technical assistance specialist reviewed the manual. The manual gives the facilitators an
outline to follow. The program does not have separate student workbooks to give out. The
facilitators make copies of the exercises before each session. The primary service is an open
group and runs year-round. Living in Balance is held on Mondays and Wednesdays. The program
keeps all documentation in a binder labeled LIB SPEP in the clinical director’s office. All youth
who are admitted to the program receive the primary service.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Technical assistance specialist observed the Living in Balance primary service on 3/15/2024. The
group began at 3:02 p.m. There were thirteen (13) youth, one therapist, and three staff present for
the group. The therapist was facilitating the group. The group was held in the program’s dayroom.
The technical assistance specialist utilized the manual to follow along with the lesson. The
facilitator made copies of the manual and gave them to the youth at the beginning of the chapter.
The facilitator had different youth read parts of the lesson and then had the youth answer the
student exercise questions. There were about five (5) to six (6) youth who took turns answering
the questions. The other youth were quiet during the lesson. The quiet youth did fill out the
student exercise handouts.The facilitator was knowledgeable and presented the concepts in a
clear and concise manner. The group ended at 4:03 p.m.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the program
experienced facilitator turnover. There were three (3) therapists who left the program. The three
therapists were facilitating groups at the time they left. The program ensured another trained staff
member took over facilitating the primary service. There were no gaps in service during the
annual review period.
The attendance sheet dated 3/15/2023 did not list the group end time. There were three (3)
attendance sheets which did not have any dates, or the facilitator listed. There were thirty-five (35)
dates where the program ran the primary service with more than fifteen (15) youth in a group.
There was one date in July 2023, ten (10) dates in August 2023, eleven (11) dates in September
2023, seven dates in October 2023, four dates in November 2023, one (1) date in December
2023, and one date in January 2024.
The primary service is a substance abuse curriculum and should have no more than fifteen (15)
youth in each group. The clinical director stated the program had staffing issues where the clinical
director was the only person running the cohort during the months of August 2023 and September
2023. The program had two (2) trained staff (clinical director and therapist) running groups in
October 2023 and November 2023. The clinical director stated there were more than fifteen (15)
youth in group during the December 2023 and January 2024 lesson was because the other
trained staff was off when the program ran group. The other trained therapist merged both cohorts
together for those days to ensure the youth received the service. The technical assistance
specialist informed the clinical director that there cannot be more than fifteen (15) youth in a
Substance Abuse Overlay Service (SAOS) group.
According to the clinical director, the program’s practice guidelines in the event of turnover is one
of the other trained facilitators on staff will continue with the group cohort.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows there were
thirteen (13) fidelity monitorings completed. The program was missing fidelity monitorings for the
months of February 2023 for one therapist; March 2023 for one therapist; April 2023 for the
clinical director and one therapist; May 2023 for two therapists; June 2023 for one therapist and
the clinical director; July 2023, August 2023, and October 2023 for the clinical director; and
January 2024 for one therapist. The clinical director, corporate clinical director, and one (1)
trained therapist conducted the monthly internal fidelity monitoring.
The fidelity monitoring completed on 2/20/2023 did not list the observer on the front of the fidelity
form, but there was the observer signature on the back. One (1) of the fidelity forms dated
3/20/2023 has the clinical director’s signature as the observer and the other fidelity form dated
3/20/2023 has the therapist’s signature as the observer. The clinical director stated she and the
therapist co-facilitated the group and conducted a fidelity on each other. The fidelity form on
6/5/2023 did not have the date listed on the front of the form. The date was located on the back
with the signatures.
According to the clinical director, the clinical director monitors each group monthly and provides
SPEP fidelity checklist and status report for documentation. The clinical director discusses the
fidelity check with the facilitator upon completion.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the
program did not identify corrective actions on their monthly internal fidelity monitorings. The
clinical director will discuss any identified corrective actions with the facilitator and will re-evaluate
during next month’s fidelity check.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the program
completed annual performance evaluations for three (3) therapists. The evaluations have an area
to document the facilitator’s skill delivering the primary service and all three evaluations contain
information specific to the therapist’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 24 youth in the
sample, 67% (16 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 24 youth in the
sample, 92% (22 of 24) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 63%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 7 youth
Moderate-High = 8 youth
High = 0 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 24 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 63% (15 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 0% (0 of 24) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 5 25 20%
Totals 69 95 73%
This SPEP report evaluates Living In Balance, an intervention delivered at Alachua Academy.
31 youth completed the program during the review period. Of those youth, 24 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 24 total youth sampled, only
17 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 12
and 46 weeks of service, with an average of 34 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, all 24 included dosage in the EBS Module. Of those youth with correct dosage, 22
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
received at least the recommended hours of service. Youth in the sample completed between 33
and 330 hours of service, with an average of 90 hours.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 94%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups and fidelity monitoring is completed every month for
each facilitator.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

report_type: SPEP

file_name: FY2023-24 Alachua Academy LIB.pdf

program_name_raw: Alachua Academy

program_status_note: Open

service_type: Living in Balance


state: FL

facility_name: Alachua Academy

summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Alachua Academy
Wayne Halfway House
(Contract Provider)
3430 NE 39th Avenue
Gainesville, Florida 32609
Primary Service: Impact of Crime (IOC)
SPEP Review Period: February 1, 2023-January 31, 2024
SPEP Review Conducted: 3/13/2024
Florida Department of Juvenile Justice
Report Date: 7/19/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the
program used Impact of Crime (IOC) as a primary service. Seven (7) staff were trained in the
primary service. The trained staff were three (3) case managers, the recreation therapist, the
transitional specialist, a PRN (as needed) senior case manager, and the assistant facility
administrator. The training is a three-day, 24-hour training currently being provided online via
Microsoft Teams. The documentation showed the recreation therapist, three (3) case managers,
and the transitional specialist conducted the groups. The trained staff members who conducted
the monthly internal fidelity monitoring were the PRN senior case manager, and two (2) case
managers. A trained staff member can conduct the monthly fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Documentation for 2/1/2023 through 1/31/2024 shows the program uses the Impact of Crime:
Addressing the Harm v 4.0 as the manual. Technical assistance specialist reviewed the manual.
The facilitator’s manual is divided into seven chapters, an appendix with handouts, and the
student guide section (which includes the student exercises). The manual gives the facilitators an
outline to follow. The facility does not have separate student workbooks to give out. The
facilitators make copies of the exercises before each session. The group is a closed group with no
more than twelve youth in each cohort. The curriculum is to be delivered to the youth over a
minimum of sixteen (16) weeks for thirty-two (32) hours. The program runs the primary service on
Tuesdays and Thursdays. The program keeps all documentation in a binder labeled IOC SPEP in
the clinical director’s office.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Technical assistance specialist observed the Impact of Crime primary service on 3/13/2024. The
group began at 4:22 p.m. There were seven (7) youth, the facilitator, and one staff member
present during the group. The group was held in one of the program’s classrooms. The facilitator
provided the technical assistance specialist with a copy of the lesson. The facilitator followed the
lesson and read the lesson verbatim. The facilitator ensured the youth understood the lesson
concepts and provided clarification when necessary. The facilitator was knowledgeable of the
material. The youth demonstrated their understanding of the Impact of Crime (IOC) concepts
during their group discussion. The youth actively participated in group. The group ended at 5:20p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the program
experienced facilitator turnover. The program had one case manager, the senior case manager
(PRN), the recreation therapist leave the program. The program conducted six cohorts. The dates
of the cohorts were 1/10/2023-8/10/2023, 1/19/2023-unknown, 5/5/2023-5/30/2023, 5/11/2023-
8/11/2023, 12/20/2023-current (as of 1/31/2024), and 1/31/2024-current (as of 1/31/2024). There
was a gap in service between 5/2/2023 and 6/13/2023 during the cohort which began on
1/10/2023. There was a gap in service between 3/31/2023 and 4/18/2023 and another gap
between 5/23/2023 and 6/8/2023 during the cohort which began on 1/19/2023. There was a
fifteen (15) day gap in service between 7/18/2023 and 8/2/2023 during the cohort beginning on
5/11/2023. There was a gap in service between the cohort which ended on 8/11/2023 and the
cohort which began on 12/20/2023.
The cohort which began on 1/10/2023 began with five (5) youth and ended with no youth
completing the service. One youth switched group on 1/17/2023. The attendance sheet dated
2/23/2023 has a note dated 5/11/2023 which states group time was added to the attendance
sheet based on the program’s shift log. A note on the attendance sheet dated 3/9/2023 (note
dated 5/11/2023) stating the clinical director was unable to verify group time in shift logbook. The
attendance sheet shows the group started at 7:03p and ended at 7:05p. One youth discharged
from the program on 3/14/2023. An attendance sheet dated 3/30/2023 did not list the group times.
One youth was discharged on 4/10/2023. The attendance sheet dated 4/10/2023 did not have any
youth signatures. The group was run back-to-back-to-back on 4/10/2023, 4/11/2023, and
4/12/2023. One youth was released on 4/21/2023. The attendance sheet dated 6/15/2023 does
not have youth signatures. The attendance sheet dated 6/27/2023 does not have youth
signatures and does not list the group times; a note says the group didn't run because the
facilitator was absent. The attendance sheet dated 6/29/2023 shows the same note. There was a
gap in service from 6/22/2023 to 7/17/2023 and then another gap in service from 7/21/2023 to
8/1/2023. The last attendance sheet dated 8/10/2023 shows the facilitator was on chapter seven
objective three. There are five total objectives to be completed during chapter seven. There were
no additional attendance sheets provided to document the youth receiving the final two (2)
objectives.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The cohort which began on 1/19/2023 began with six (6) youth and ended with no youth
completing the service. The attendance sheet dated 2/9/2023 shows the facilitator added a box
for objective three, but chapter two only has two objectives. One youth was released on
2/14/2023. The attendance sheet dated 2/16/2023 shows a drawn in box for chapter two objective
four (again no such objective). The facilitator again added an objective to the attendance sheet
dated 5/2/2023. There were no other attendance sheets provided to determine if the youth
completed the cohort. The date of the last attendance sheet was 6/30/2023. There was a note on
a blank attendance sheet which stated that due to unforeseen circumstances, we are unable to
complete this cohort.
The cohort which began on 5/5/2023 began with nine (9) youth and ended with no youth
completing the service. The case manager who began the cohort left the program on 6/5/2023.
The program did not transition the youth to another trained facilitator. The clinical director stated a
case manager and the recreation therapist told her the group needed to end and start a new
cohort since the cohort did not have a co-facilitator. The technical assistance specialist informed
the Clinical Director the information she received was incorrect. The program could transition the
youth to another trained facilitator. Transitioning the youth to another trained facilitator is better
than stopping the cohort abruptly. The technical assistance specialist stated the cohort could
continue by having a trained facilitator take over and conduct a review of the content for the youth.
The facilitator could then progress through the rest of the service once the youth completed the
review. The program did not resume the cohort.
The cohort which began on 5/11/2023 began with seven (7) youth and ended with zero (0) youth
completing the service. There was a note stating that due to staff shortage, this curriculum was
could not be completed. The Impact of Crime (IOC) group members have been discharged.The
clinical director stated the facilitator decided to end the cohort without completing the service. The
technical assistance specialist advised the clinical director that the decision to suspend or end a
cohort should come from the clinical director or the facility administrator. The technical assistance
specialist also offered the clinical director assistance with any issues pertaining to the delivery of
the primary services.
The cohort which began on 12/20/2023 began with seven (7) youth and currently has seven (7)
youth (as of 1/31/2024).
The cohort which began on 1/31/2024 began with eight youth and currently has eight youth (as of
1/31/2024).
Technical assistance specialist spoke with the clinical director about issues with data entry into
the Evidence-Based Service (EBS) module in the Juvenile Justice Information System (JJIS). The
clinical director did not enter end dates for youth who did not complete the service and were
released from the program. Technical assistance specialist asked the clinical director to go into
EBS and update the youth’s end dates to reflect the last date the youth participated in the primary
service. The clinical director stated she did not enter those end dates because the youth did not
complete the service. The clinical director stated she thought the end dates are only entered when
the youth successfully complete the service. Technical assistance specialist stated end dates
need to be entered when the youth successfully complete the service or if the youth is released
from the program without successfully completing the service. The clinical director entered the
youth’s missing end dates into EBS.
Technical assistance specialist asked the clinical director if she used a tracker that has the
youth’s admission date and projected release date. Technical assistance specialist stated the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
clinical director could count sixteen (16) weeks back from the projected release date and use that
date as the latest the youth can be enrolled in the primary service. This would ensure the youth
was enrolled in the primary service with enough time to successfully complete the service. The
clinical director stated she would begin using a tracker.
According to the clinical director, the program’s practice guidelines in the event of turnover is one
of the other trained facilitators on staff will continue with the group cohort.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows there were
eight (8) fidelity monitorings completed. The program was missing fidelity monitorings for the
month of March 2023, May 2023, and June 2023 for one case manager; February 2023, March
2023, and May 2023 for the recreation therapist; May 2023 and June 2023 for one case manager;
and May 2023 for the transition specialist; July 2023 for the recreation therapist; August 2023 and
December 2023 for the transition specialist. There were no fidelity monitoring checklists
completed for the months of September 2023, October 2023, and November 2023 due to the
program not running any cohort during that time. The trained staff members who conducted the
monthly internal fidelity monitoring were the PRN senior case manager, and two full-time case
managers. The clinical director discusses the fidelity check with the facilitator upon completion.
The fidelity monitoring dated for 6/20/2023 did not have the facilitator listed. Technical assistance
specialist looked at the attendance sheet dated 6/20/2023 to confirm the facilitator for the fidelity
monitoring. The fidelity monitoring dated 1/31/2024 did not specify the facilitator’s name and title
or the number of youths in the group.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the
program did not identify any corrective actions on their monthly internal fidelity monitorings. The
clinical director will discuss any identified corrective actions with the facilitator and will re-evaluate
during the next month’s fidelity check.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Documentation for the annual review period of 2/1/2023 through 1/31/2024 shows the program
provided two staff evaluations to review. There were two (2) annual staff evaluations. The annual
evaluations were completed for a case manager and transition specialist. The evaluations have
an area to document the facilitator’s skill delivering the primary service. Both annual evaluations
contained detailed information about the staff’s delivery of the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 23 youth in the
sample, 30% (7 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 23 youth in the
sample, 52% (12 of 23) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 61%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 7 youth
Moderate-High = 7 youth
High = 0 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 61% (14 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 0% (0 of 23) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 5 25 20%
Totals 41 85 48%
This SPEP report evaluates Impact of Crime, an intervention delivered at Alachua Academy.
31 youth completed the program during the review period. Of those youth, 23 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 23 total youth sampled, only
20 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended weeks of service. Youth in the sample completed between 3
and 30 weeks of service, with an average of 13 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, only 20 included dosage in the EBS Module. Of those youth with correct dosage, 12
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
received at least the recommended hours of service. Youth in the sample completed between 1
and 27 hours of service, with an average of 18 hours.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 72%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Alachua Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, there are no unnecessary gaps in service, and fidelity
monitoring is completed every month for each facilitator.
Alachua Academy can optimize their SPEP Amount of Service score by ensuring that dosage for
all youth is recorded accurately in EBS and by ensuring that youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

source: DJJ

file_name: FY2023-24 Alachua Academy IOC.pdf

service_type: Impact of Crime