summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC.
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Male Healthy Relationships
SPEP Review Period: March 15, 2020-March 14, 2021
SPEP Review Conducted: April 19, 2021
Florida Department of Juvenile Justice
Report Date: 5/26/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2020 through March 14,
2021 showed the program utilizes Male Healthy Relationships bundle as a primary service. The
Male Healthy Relationships bundle contains two services: Teen Relationships Workbook and
Young Men’s Work. The Male Healthy Relationships bundle training is an in-service training
conducted by the clinical director or assistant clinical director. The documentation shows there
were four staff facilitating the primary service during the review period, but the program only
provided proof of training for the clinical director and one therapist. There were two other
therapists facilitating the primary service without any proof of training. The director of clinical
services stated the program was unable to find any documented training for two of the four
facilitators. The program decided to go ahead and conduct a current training to ensure the training
was completed and documented. The clinical director conducts the monthly internal fidelity
checks.
According to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitators on the job
training plan. All facilitators will receive training prior to conducting the service.
The program can earn 1 point if all facilitators receive formal training prior to delivering services to
youth.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Teen Relationship Workbook and the Young Men's Work manual. The
Teen Relationships Workbook is broken down into six sections. The first section is evaluating your
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
relationship. Section two is understanding abuse. Section three is social influences. Section four
is building healthy relationships. Section five is making good decisions. Section six is the review.
There are sixty-eight reproducible worksheets given to the youth during the sessions. Each lesson
has information on how the facilitator should present the material. There are no separate student
workbooks. The facilitators make copies of the current lessons to hand out to the youth. The
Young Men's Work manual is broken down into twenty-six sessions. Each session has at least
one student exercise in them. There are a total of thirty-eight exercises during the group. The
manual provides information for the facilitator to conduct each session.
The program provides the Male Healthy Relationship bundle on Thursdays and Sundays for one
hour each session. The program keeps all SPEP documents in a binder labeled Male Healthy
Relationships. The program states all youth admitted to the program are scheduled to complete
the primary service. The group is an open group.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Male Healthy Relationships was observed on 4/8/2021 via Microsoft Teams. The
group began at 3:03p. There were twelve youth in attendance. The youth began with the check in.
The youth stated their high for the day and their low for the day. The lesson was on gender roles.
The facilitator asked the youth how gender roles play a part of abusive relationships. The
facilitator then moved onto the exercise Man Box. The facilitator asked the youth to write down
what makes someone a man and put that in the box. There were several youth holding side
conversations during the group. The facilitator had to constantly redirect the youth. The youth
were then asked to state a positive impact and a negative impact a male and female had on their
life. Group ended at 3:58p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period shows the program had facilitator
turnover which adversely affected the delivery of Male Healthy Relationships. The program had
two trained therapists and one clinical director who facilitated the primary service during the
annual review period. The program had another trained therapist come from another program to
assist in the delivery of the primary service when the clinical director and licensed therapist left the
program. The non-licensed therapist is the only facilitator left at the program. The group is an
open group run year-round.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the facility's Standard Program Evaluation Protocol procedure, Jacksonville Youth
Academy utilizes a Probationary Review Status Process designed for effective management of
risk through individualized coaching, training and verification activities of all new employees
during their first 90 days. The process encourages personal development and recognizes areas of
improvement thus reducing staff turnover. In addition, the program provides positive incentives, in
an effport to improve employee motivation and dedication and lead to better employee retention.
Through this policy employees receive credit for years of service, employment referrals, and for
contributing to the positive culture, teamwork environment and for going above and beyond
normal job expectations. Each employee who receives credit for doing these things is entered in a
monthly drawing at each facility. In the event of facilitator turnover, the co-facilitator will assume
the role of the facilitator for the group. If a co-facilitator is not present, a new qualified facilitator for
the primary service will take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period showed there were twenty-five fidelity
monitorings completed. The fidelity check dated 8/28/2020 shows the clinical director was the
facilitator and the observer, but the signature on the back page was a different person. The
clinical director conducts the monthly internal fidelity monitorings.
The program was missing fidelity checks for the following months: March 2021 for one therapist,
February 2021 for the clinical director and one therapist, January 2021 for the clinical director,
December 2020 for the clinical director, November 2020 for the clinical director and one therapist,
October 2020 for the clinical director, September 2020 for the clinical director and one therapist,
July 2020 for the clinical director, May 2020 for the clinical director, and April 2020 for the clinical
director
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to documentation, there were no corrective actions noted during the fidelity monitoring.
According to the facility's Standard Program Evaluation Protocol procedure, based on the results
of the monthly fidelity monitoring corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event that an issue identified during the primary service is detrimental to the
youth’s safety and/or treatment, the matter will be taken up directly with the program’s Human
Resource Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
According to the documentation provided, evaluations were completed on the three trained
facilitators. The evaluations had an area to note the facilitator's performance in the evidence-
based programming; however, there was not any specific information about the facilitator’s skill in
delivering the primary service. The evaluations only have a numerical value assigned along with
listing the primary service the staff is facilitating.
The facility's Standard Program Evaluation Protocol states, an annual evaluation of performance
will be conducted on all the facilitators of each primary service. The performance evaluation is
comprised of feedback from the fidelity monitoring tools as well as an assessment of the
facilitator’s skill in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 31 youth in the
sample, 13% (4 of 31) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 40 contact hours for this type of service. Of the 31 youth in the
sample, 39% (12 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 12 youth
High = 14 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (29 of 31) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (14 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 65 95 68%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Jacksonville
Youth Academy.
33 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 31 total youth sampled, only
10 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 4
received at least the recommended weeks of service. Youth in the sample completed between 6
and 34 weeks of service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 2 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 12 included dosage in the EBS Module. Of those youth with correct dosage, 12
received at least the recommended hours of service. Youth in the sample completed between 44
and 78 hours of service, with an average of 72 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring all facilitators receive formal training prior to delivering services to youth, facilitator
turnover does not impact groups, fidelity monitoring is completed every month for each facilitator,
and each facilitator’s performance evaluation includes discussion of their skill in delivering this
primary service.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

facility_name: Jacksonville Youth Academy

summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Cannabis Youth Treatment
SPEP Review Period: 3/15/2021-3/14/2022
SPEP Review Conducted: 3/29/2022
Florida Department of Juvenile Justice
Report Date: 7/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2021 through March 14,
2022 showed the program utilizes Cannabis Youth Treatment (CYT) as a primary service. CYT
training is an in-service training provided by the clinical director or other approved person. The
documentation shows there were five staff trained to facilitate CYT during the annual review
period- the former clinical director, the current clinical director, a licensed therapist, and two non-
licensed therapists. The licensed therapist and current clinical director facilitated the primary
service during the annual review period. The current clinical director, the licensed therapist, a non-
licensed therapist, and the regional clinical director conducted the internal fidelity checks for the
annual review period.
Pursuant to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitator's on the job
training plan. The facilitator will receive additional training on the curriculum as needed from the
licensed clinician documented within the clinical supervision. All facilitators will receive training
prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Cannabis Youth Treatment Series, volumes one and two. The manuals
provide step by step procedures for implementing the program. There are no student manuals.
The facilitators make copies of the worksheets and pass them out to the youth during group. The
first two lessons are conducted as individual sessions. These sessions are an hour in length.
Lessons three through twelve are conducted as a group.
All youth are required to receive the primary service. The program maintains all documentation in
a binder labeled Cannabis Youth Treatment (CYT), located in the clinical director’s office. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
binder contains all attendance sheets, internal fidelity checklists, and training certificates. The
program provides CYT one time per week on Tuesdays for one hour. CYT is run as a closed
group.
Pursuant to the facility's Standardized Program Evaluation Protocol procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Cannabis Youth Treatment was observed on 3/29/2022 via Teams. The group began
at 2:37pm and was held in the program’s cafeteria. There were ten youth, one facilitator, the
clinical director, and a youth care worker present. The facilitator was knowledgeable of the
material and followed the lesson plan. The facilitator discussed anger and triggers with the youth.
The youth had to complete an activity from the workbook. The group ended at 3:40p.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 15, 2021 through March 14,
2022 shows the program had facilitator turnover which adversely affected the delivery of
Cannabis Youth Treatment (CYT). The former clinical director and two unlicensed therapists left
the program during the annual review period.
The program ran four cohorts during the annual review period. The dates of the cohorts are
4/29/2021-7/12/2021, 7/26/2021-10/24/2021, 10/24/2021-1/26/2022, and 2/7/2022-current. There
was a gap in service between the cohort ending December 2020 and the cohort beginning
4/29/2021. The compliance director stated the company did not renew the 397 license. Without
the 397 license, the program could not use an unlicensed therapist as a facilitator for the primary
service. The former clinical director left the program in February 2021. The current director of
clinical services received training in the service in April 2021. The program did not have any
licensed therapists until the current clinical director was hired. The compliance director also stated
all youth at the program received the primary service prior to the gap. This information was
confirmed with a review of data in the Evidence-Based Services module of the Juvenile Justice
Information System. Newly admitted youth during the gap were added to the cohort which began
on 4/29/2021.
The 4/29/2021 cohort started with nine youth and ended with eight youth. The 7/26/2021 cohort
started with four youth and ended with three youth. The 10/24/2021 cohort started and ended with
the same ten (10) youth. The 2/7/2022 cohort began with ten (10) youth and currently has ten (10)
youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the facility's Standard Program Evaluation Protocol procedure, TrueCore utilizes a
Probationary Review Status Process designed for effective management of risk through
individualized coaching, training and verification activities of all new employees during their first
90 days. The process encourages personal development and recognizes areas of improvement in
an effort to reduce staff turnover. In addition, the program provides positive incentives to improve
employee motivation and dedication and lead to better employee retention. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of March 15, 2021 through March 14,
2022 shows the program conducted eleven monthly internal fidelity checks. The clinical director,
the licensed therapist, and the regional clinical director conducted the fidelity checks. There were
no missing internal fidelity checks during the annual review period. The program conducts the
fidelity monitoring during lessons three through twelve since those lessons are conducted as a
group. Lessons one and two are conducted as individual sessions.
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on the issue by a qualified facilitator in the primary service. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event an issue identified during the primary service is detrimental to the youth’s
safety and/or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an evaluation for an unlicensed therapist, licensed therapist (former
director of clinical services), and the current clinical director. The evaluation had an area to note
the facilitator's performance in delivering Cannabis Youth Treatment. Two of the three evaluations
had specific information about the facilitator’s skill delivering the primary service. The former
director of clinical services' annual evaluation did not contain specific information related to their
delivery of the primary service. The former director of clinical services did not regularly facilitate
groups, but was assisting the program with delivering the primary service since the program was
dealing with facilitator turnover during the annual review period.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, an annual evaluation
of performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 12 weeks for this type of service. Of the 24 youth in the
sample, 67% (16 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 12 contact hours for this type of service. Of the 24 youth in the
sample, 92% (22 of 24) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 14 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 58%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 24 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (22 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 58% (14 of 24) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 92 100 92%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Jacksonville
Youth Academy.
24 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 6 points for Amount of Service: Duration. Of the 24 total youth sampled, all
24 included dosage with end dates in the EBS Module. Of those youth, 16 received at least the
recommended weeks of service. Youth in the sample completed between 6 and 27 weeks of
service, with an average of 13 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, all 24 included dosage in the EBS Module. Of those youth, 22 received at least the
recommended hours of service. Youth in the sample completed between 6 and 15 hours of
service, with an average of 13 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

file_name: FY2021-22 JYA CYT rev220714.pdf

program_name_raw: Jacksonville Youth Academy - TrueCore

program_status_note: Closed

service_type: Cannabis Youth Treatment


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Impact of Crime
SPEP Review Period: 3/15/2021-3/14/2022
SPEP Review Conducted: 3/29/2022
Florida Department of Juvenile Justice
Report Date: 7/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2021 through March 14,
2022 showed the program utilizes Impact of Crime (IOC) as a primary service. IOC training is
three days, provided by a qualified trainer. The documentation shows there were six staff trained
to facilitate Impact of Crime during the annual review period- the former director of case
management/transition manager, one administrative assistant, the clinical director, a licensed
therapist, and two case managers. Two case managers and the administrative assistant facilitated
the primary service during the annual review period. The director of case management/transition
manager, a case manager, the licensed therapist, and the clinical director conducted the internal
fidelity checks for the annual review period.
Pursuant to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitator's on the job
training plan. The facilitator will receive additional training on the curriculum as needed from the
licensed clinician documented within the clinical supervision. All facilitators will receive training
prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm manual. The manual opens with
an introduction to the Impact of Crime (IOC) curriculum. Then the manual goes over all seven
chapters of the service and provides a script for the facilitator to follow. The manual also has an
appendix and a student guide with all the student exercises. The program makes copies of the
student exercises and passes them out to the youth during each lesson.
The program conducts IOC on Tuesdays and Thursdays. The program keeps all SPEP
documentation (attendance sheets, fidelity checks, and training certificates) in a binder labeled
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
"IOC SPEP documentation" in the case management office. All youth admitted to the program are
required to participate in IOC. The primary service is a closed group with no more than twelve
youth assigned to one cohort.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 3/24/2022 via Teams. The group began at 6:12pm
and was held in the program’s cafeteria. There were ten (10) youth, one facilitator, and the clinical
director present for the group. The facilitator was knowledgeable of the material and followed the
lesson plan. The facilitator showed the “Life Sentence” videos to the youth. The facilitator stopped
the video after each clip and asked the youth what the major impact areas were. The youth
watched the videos without any disruption. The group ended at 7:12p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 15, 2021 through March 14,
2022 shows the program had facilitator turnover which adversely affected the delivery of Impact of
Crime. The director of case management/transition manager and a case manager left the
program during the annual review period. The program ran five cohorts during the annual review
period. The dates of the cohorts are 2/8/2021-5/5/2021, 4/20/2021-8/5/2021, 8/17/2021-
12/8/2021, 11/9/2021-2/24/2022, and 2/15/2022-current. There were no gaps in service.
The 2/8/2021 cohort began with ten (10) youth and ended with three youth. Six youth were
discharged and one youth was arrested prior to completing the primary service. The attendance
sheets show one case manager being added as the co-facilitator on 2/22/2021, but that case
manager did not facilitate any of the groups.
The 4/20/2021 cohort began six youth and ended with seven youth. A youth was added to the
cohort prior to the cohort reaching chapter two, as permitted by the Department during the
COVID-19 pandemic. The youth was admitted to the program shortly after the cohort began. The
youth was provided supplemental sessions to catch the youth up to the cohort. The program did
not have any other youth in the program who needed the primary service, so the case manager
and director of case management contacted the technical assistance specialist to see if the youth
could be added to the cohort. The technical assistance specialist stated the youth could be added
as long as the facilitator conducted individual sessions with the youth to ensure the youth received
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
the lessons they missed. The attendance sheets document the facilitator covered the first chapter
for the new youth on 4/26/2021 in two hours.
The 8/17/2021 cohort started with five youth and ended with six youth. Two youth were added to
the group prior to the cohort completing chapter one. One youth was arrested on 9/15/2021. The
cohort began with the administrative assistant facilitating the cohort. The director of case
management/transition manager stated the administrative assistant was trained while she was a
youth care worker, but transitioned to the administrative assistant position. The director of case
management was informed the administrative assistant is unable to facilitate the cohort due to the
contract only allowing the case management staff and youth care workers to facilitate the primary
service. The attendance sheets showed a case manager was added as a co-facilitator on
8/27/2021. On 9/13/2021, the director of case management/transition manager took over the
group and facilitated by himself. On 10/21/2021, the case manager who was listed as the co-
facilitator came back and began co-facilitating with the director of case management/transition
manager.
The 11/9/2021 cohort began with eight youth and ended with seven youth. One youth was
arrested on 1/13/2022.
The 2/15/2022 cohort began with ten (10) youth and currently has ten (10) youth.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, TrueCore utilizes a
Probationary Review Status Process designed for effective management of risk through
individualized coaching, training and verification activities of all new employees during their first
90 days. The process encourages personal development and recognizes areas of improvement
thus reducing staff turnover. In addition, the program provides positive incentives, in an attempt to
improve employee motivation and dedication, leading to better employee retention. Through this
policy employees receive credit for years of service, employment referrals, for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of March 15, 2021 through March 14,
2022 shows the program conducted sixteen monthly internal fidelity checks for facilitators of
Impact of Crime, accounting for each facilitator each month services were provided. The director
of case management/transition manager, a case manager, the licensed therapist, and the clinical
director conducted the fidelity checks. There were no missing internal fidelity checks during the
annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for the primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring for facilitators of Impact of Crime.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on the issue by a qualified facilitator. The additional coaching and/or
supervision will be documented by the person conducting the fidelity monitoring in a note and
placed in the clinical supervision file. If during additional fidelity monitoring the previously identified
area addressed continues to be an issue, the next stage is completing a written warning and
placing the facilitator on a Performance Improvement Plan. The Performance Improvement Plan
will state the issue identified, the tasks the facilitator needs to complete to correct the issue, and a
target date for completion. The plan will then be reviewed with the facilitator by the qualified
facilitator conducting the monitoring, and the facilitator’s supervisor. In the event that an issue
identified during the primary service is detrimental to the youth’s safety and/or treatment, the
matter will be taken up directly with the program’s Human Resource Manager and be addressed
in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an evaluation for the former director of case management/transition
manager and two case managers. The evaluation had an area to note the facilitator's
performance in facilitating Impact of Crime. Two of the three evaluations had specific information
about the facilitator’s skill delivering the primary service. The former director of case
management’s annual evaluation did not contain specific information related to their delivery of
the primary service.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, an annual evaluation
of performance will be conducted on all facilitators of each primary service. The performance
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 24 youth in the
sample, 58% (14 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 24 youth in the
sample, 100% (24 of 24) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 14 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 58%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 100 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (22 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 58% (14 of 24) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 77 85 91%
This SPEP report evaluates Impact of Crime, an intervention delivered at Jacksonville Youth
Academy.
24 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 24 total youth sampled, all
24 included dosage with end dates in the EBS Module. Of those youth, 14 received at least the
recommended weeks of service. Youth in the sample completed between 6 and 20 weeks of
service, with an average of 14 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, all 24 included dosage in the EBS Module. Of those youth, 24 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
recommended hours of service. Youth in the sample completed between 32 and 40 hours of
service, with an average of 33 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Impact of Crime is an intervention suitable for all youth. The Department recommends providing
this service to every youth in the program.
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups and each facilitator’s performance evaluation
includes discussion of their skill in delivering this primary service.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2021-22 JYA IOC.pdf

service_type: Impact of Crime


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Male Healthy Relationships
SPEP Review Period: 3/15/2021-3/14/2022
SPEP Review Conducted: 3/29/2022
Florida Department of Juvenile Justice
Report Date: 7/6/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2021 through March 14,
2022 showed the program utilizes the Male Healthy Relationships bundle as a primary service.
The Male Healthy Relationships bundle consists of two services: The Teen Relationships
Workbook and Young Men’s Work. The program has a written training protocol for both Young
Men’s Work and Teen Relationships. The Male Healthy Relationships bundle training is an in-
service training conducted by the director of clinical services, clinical director, or assistant clinical
director. The documentation shows there were seven staff trained in the primary service- three
unlicensed therapists, the recreation therapist, the current clinical director, the former clinical
director, and a licensed therapist (who was the former director of clinical services). Six of the
seven trained staff conducted the groups during the annual review period. The current clinical
director, the licensed therapist, an unlicensed therapist, and the recreation therapist conducted
the monthly internal fidelity checks during the annual review period.
Pursuant to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitator's on the job
training plan. The facilitator will receive additional training on the curriculum as needed from the
licensed clinician documented within the clinical supervision. All facilitators will receive training
prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes The Teen Relationships Workbook and the Young Men's Work manual. The
program maintains the attendance sheets, fidelity checklists, and training certificates in the SPEP
binder labeled "Male Healthy Relationships bundle" in the clinical director’s office. The Teen
Relationships Workbook is broken down into six sections with sixty-eight (68) reproducible
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
worksheets given to the youth during the sessions. Each lesson has information on how the
facilitator should present the material. There are no separate student workbooks.
According to the program’s Gender-Specific SPEP Service “Bundle Proposal”, Teen Relationships
component assists youth to develop healthy relationships and prevent dating abuse and domestic
violence. Teen Relationships helps teens identify healthy and unhealthy characteristics of
relationships and to understand the dynamics of emotional, physical, and sexual abuse in intimate
relationships; youth explore and clarify their own values and attitudes about gender roles and be
empowered to make healthy decisions about their own relationships.
The Young Men's Work manual is broken down into twenty-six (26) sessions. Each session has at
least one student exercise in them. There are a total of thirty-eight (38) exercises during the
group. The manual provides information for the facilitator to conduct each session.
According to the program's Gender Specific SPEP Service "Bundle Proposal", Young Men’s Work
is a gender-specific component that teaches problem solving and violence prevention with
gender-identity topics ranging from power, violence, bullying, anger, and frustration, while also
addressing healthy relationships with women and interpersonal relationships between males.
Additionally, Young Men’s Work enhances social skills to cope with negative influences from
others and the environment that are a large part of the risk in youth transitioning back to
disadvantaged neighborhoods.
Young Men's Work and Teen Relationships are delivered one time per week and are run as open
groups throughout the youth's length of stay. The youth will participate in the components until a
minimum of 40 combined contact hours are reached. The program runs the bundle services on
Thursdays and Mondays. The documentation shows the group was run continuously throughout
the review period.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of the Teen Relationships group was observed on 3/24/2022. The group began at
2:36pm and was held in the program’s cafeteria. There were ten (10) youth, one facilitator, and
one staff present during the group. The facilitator was knowledgeable of the material and followed
the lesson plan. Several of the youth were holding side conversations during the group. The
facilitator did her best to redirect the youth. The facilitator was able to get the youth to answer the
questions related to the service. The lesson dealt with ending or continuing difficult relationships.
The group ended at 3:42pm.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 15, 2021 through March 14,
2022 shows the program did have facilitator turnover which adversely affected the delivery of
Male Healthy Relationships (MHR). The former clinical director and two unlicensed therapists left
the program while facilitating groups. The program did not have any gaps in service as MHR is
run year-round. The youth were transitioned to another therapist to ensure continued delivery of
the primary service. The program runs Teen Relationships on Thursdays and Young Men’s Work
on Mondays.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, TrueCore utilizes a
Probationary Review Status Process designed for effective management of risk through
individualized coaching, training and verification activities of all new employees during their first
90 days. The process encourages personal development and recognizes areas of improvement in
an effort to reduce staff turnover. In addition, the program provides positive incentives to improve
employee motivation and dedication, leading to better employee retention. Through this policy
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Documentation reviewed for the annual review period of March 15, 2021 through March 14, 2022
shows the program conducted twenty-seven (27) monthly internal fidelity checks for facilitators of
Male Healthy Relationships. A trained therapist, clinical director, or director of clinical services
conducts the internal fidelity checks. The non-licensed therapist did not have an internal fidelity
check completed for her for the month of June 2021 because she abruptly left the program on
6/10/2021. The program was missing fidelity monitorings for the following months: June 2021 for
two therapists, August 2021 for the clinical director, and October 2021 for one therapist.
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for the primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring of facilitators of Male Healthy Relationships.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, based on the results of
monthly fidelity monitoring corrective action may be needed. The first stage of addressing areas
identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on the issue by a qualified facilitator. The additional coaching and/or
supervision will be documented by the person conducting the fidelity monitoring in a note and
placed in the clinical supervision file. If during additional fidelity monitoring the previously identified
area addressed continues to be an issue, the next stage is completing a written warning and
placing the facilitator on a Performance Improvement Plan. The Performance Improvement Plan
will state the issue identified, the tasks the facilitator needs to complete to correct the issue, and a
target date for completion. The plan will then be reviewed with the facilitator by the qualified
facilitator conducting the monitoring, and the facilitator’s supervisor. In the event an issue
identified during the observation is detrimental to the youth’s safety and/or treatment, the matter
will be taken up directly with the program’s Human Resource Manager and be addressed in
accordance with the employee handbook.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an evaluation for an unlicensed therapist, licensed therapist (former
director of clinical services), and the current clinical director. The evaluation had an area to note
the facilitator's performance in providing Male Healthy Relationships (MHR). Two of the three
evaluations had specific information about the facilitator’s skill delivering the primary service. The
former director of clinical services' annual evaluation did not contain specific information related to
their delivery of MHR. The former director of clinical services was assisting the program with
delivering MHR since the program was dealing with facilitator turnover during the annual review
period.
Pursuant to the facility's Standard Program Evaluation Protocol protocol, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 24 youth in the
sample, 100% (24 of 24) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 24 youth in the
sample, 100% (24 of 24) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 6 youth
High = 14 youth
Total Youth in Sample = 24 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 58%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 24 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (22 of 24) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 58% (14 of 24) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 93 95 98%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Jacksonville
Youth Academy.
24 youth completed the program during the review period. Of those youth, all 24 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 24 total youth sampled,
only 24 included dosage with end dates in the EBS Module. Of those youth with correct dosage,
24 received at least the recommended weeks of service. Youth in the sample completed between
26 and 63 weeks of service, with an average of 33 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program earned 10 points for Amount of Service: Contact Hours. Of the 24 total youth
sampled, only 24 included dosage in the EBS Module. Of those youth with correct dosage, 24
received at least the recommended hours of service. Youth in the sample completed between 46
and 118 hours of service, with an average of 63 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups and each facilitator’s performance evaluation
includes discussion of their skill in delivering this primary service.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

file_name: FY2021-22 JYA MHR.pdf

service_type: Male Healthy Relationships