summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Cannabis Youth Treatment
SPEP Review Date(s): 3/9/2020
Florida Department of Juvenile Justice
Report Date(s): 3/9/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation showed there were three staff trained to facilitate Cannabis Youth
Treatment during the annual review period. The training is an in-service training provided by the
clinical director or other approved person.
According to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitator's on the job
training plan. All facilitators will receive training prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Cannabis Youth Treatment Series Volumes one and two. The manuals
provide step by step procedures for implementing the program. There are no student manuals.
The facilitators make copies of the worksheets and pass them out to the youth during group.
Pursuant to the facility's Standardized Program Evaluation Protocol (SPEP) procedure, each
primary service will be facilitated in accordance to its manual. The primary service will be
conducted as indicated in the manual/protocol; this includes adherence to the session plan, group
size, population, frequency, needed materials, duration, and interventions. The facilitator of the
primary service will specify the topic covered, interventions utilized, duration of the service, youth
present, and progress for each session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
A session of Cannabis Youth Treatment was unable to be observed. The program is currently not
running the primary service group. The program doesn’t have enough trained facilitators to run
groups.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the review period shows the program had facilitator turnover
which adversely affected groups. Groups were run according to the facility's schedule. There were
three cohorts noted during the review period. The dates of the cohorts were 3/6/2019-5/8/2019,
6/5/2019-8/28/2019, and 10/16/2019-12/18/2019. According to the facility administrator, the
program has had a current gap of over one month because there are currently no trained
facilitators for the primary service group and the census is down. The program had an admission
freeze in July 2019-August 2019. The majority of the program’s youth have already completed
Cannabis Youth Treatment. The facility administrator stated the program will begin the primary
service groups once there are trained facilitators available to run the groups.
Pursuant to the facility's operating procedure, Jacksonville Youth Academy utilizes a Probationary
Review Status Process designed for effective management of risk through individualized
coaching, training and verification activities of all new employees during their first 90 days. The
process encourages personal development and recognizes areas of improvement in an effort to
reduce staff turnover. In addition, the program provides positive incentives in an effort to improve
employee motivation and dedication and lead to better employee retention. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period showed there were three fidelity
monitorings completed for Cannabis Youth Treatment. The fidelity monitoring took place on
4/17/2019, 5/8/2019, and 6/19/2019. There was no other documentation provided for the other
months in the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
According to the facility's operating procedure, designated staff are responsible for conducting
fidelity monitoring of each facilitator of a primary service on a monthly basis. The staff conducting
the fidelity monitoring has received formal training by a qualified trainer in the primary service.
Fidelity monitoring is documented on a fidelity monitoring tool developed specifically for that
primary service by the developer of the manual, or an internal fidelity checklist created for
manualized or non-manualized curriculum delivered in the program. The staff performing the
fidelity monitoring of the primary service is verifying the facilitator is conducting the group in
accordance with the manual/protocol. This includes verifying the facilitator adheres to the session
plan, group size, population, frequency, needed materials, duration, and interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring.
Pursuant to the facility's Standardized Program Evaluation Protocol procedure, based on the
results of the monthly fidelity monitoring, corrective action may be needed. The first stage of
addressing areas identified as needing improvement through fidelity monitoring is by providing
coaching and supervision to the facilitator on that issue by a qualified facilitator in the primary
service. The additional coaching and/or supervision will be documented by the person conducting
the fidelity monitoring in a note and placed in the clinical supervision file. If, during additional
fidelity monitoring, the previously identified area addressed continues to be an issue, the next
stage is completing a written warning and placing the facilitator on a Performance Improvement
Plan. The Performance Improvement Plan will state the issue identified, the tasks the facilitator
needs to complete to correct the issue, and a target date for completion. The plan will then be
reviewed with the facilitator by the qualified facilitator conducting the monitoring, and the
facilitator’s supervisor. In the event an issue identified during the primary service is detrimental to
the youth’s safety and/or treatment, the matter will be taken up directly with the program’s Human
Resource Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
According to documentation, annual evaluations were completed on the trained facilitators. The
evaluations had an area to note the facilitator's performance in the evidence-based programming.
The facility's Standardized Program Evaluation Protocol procedure states, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 12 weeks for this type of service. Of the 29 youth in the
sample, 55% (16 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 7 youth
High = 19 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 66%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 19 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (29 of 29) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 66% (19 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 84 100 84%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Jacksonville
Youth Academy.
33 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Duration. Of the 29 total youth sampled, only
27 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 16
received at least the recommended weeks of service. Youth in the sample completed between 11
and 14 weeks of service, with an average of 13 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, all 29 included dosage in the EBS Module. Of those youth, 29 received at least the
recommended hours of service. Youth in the sample completed between 12 and 17 hours of
service, with an average of 12 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups and by ensuring fidelity monitoring is
completed every month for each facilitator.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Jacksonville Youth Academy

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Impact of Crime
SPEP Review Date(s): 3/9/2020
Florida Department of Juvenile Justice
Report Date(s): 3/9/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation showed there were six staff trained to facilitate Impact of Crime
during the annual review period. There are currently three staff left at the program who are trained
facilitators. The training is provided by a qualified trainer over four days.
Pursuant to the facility's operating procedure, the facilitator for the primary service will have
training from a qualified trainer. This training will be documented in the facilitator's on the job
training plan. All facilitators will receive training prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Impact of Crime: Addressing the Harm v 2.0. The manual provides step
by step procedures for implementing the program. There are scripts and activities for the
facilitators to use. There are no student manuals given to the youth. The facilitators make copies
of the worksheets and pass them out to the youth during group.
Pursuant the facility's Standard Program Evaluation Protocol (SPEP) procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed as a cohort recently ended on 3/2/2020,
one week prior to the annual SPEP review.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program had facilitator turnover; however, the turnover
did not adversely affect groups. There were three cohorts completed during the annual review
period. The dates of the cohorts are 3/4/2019-6/5/2019, 6/24/2019-9/23/2019, and 11/25/2019-
3/2/2020. During the 6/24/2019-9/23/2019 cohort, the original assigned facilitator had to leave the
program for personal reasons. The program transitioned the youth to another trained facilitator
without any gaps in service.
Pursuant to the facility's operating procedure, Jacksonville Youth Academy utilizes a Probationary
Review Status Process designed for effective management of risk through individualized
coaching, training and verification activities of all new employees during their first 90 days. The
process encourages personal development and recognizes areas of improvement in an effort to
reduce staff turnover. In addition, the program provides positive incentives in an effort to improve
employee motivation and dedication and lead to better employee retention. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed showed there were eleven fidelity monitorings completed. The
fidelity monitorings were completed each month with the exception of December 2019.
According to the facility's operating procedure, designated staff are responsible for conducting
fidelity monitoring of each facilitator of a primary service on a monthly basis. The staff conducting
the fidelity monitoring has received formal training by a qualified trainer in the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Fidelity monitoring is documented on a fidelity monitoring tool developed specifically for that
primary service by the developer of the manual, or an internal fidelity checklist created for
manualized or non-manualized curriculum delivered in the program. The staff performing the
fidelity monitoring of the primary service is verifying the facilitator is conducting the group in
accordance with the manual/protocol. This includes verifying the facilitator adheres to the session
plan, group size, population, frequency, needed materials, duration, and interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If, during additional fidelity
monitoring, the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event an issue identified during the primary service is detrimental to the youth’s
safety and/or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
According to documentation provided, evaluations were completed on the trained facilitators. The
evaluations had an area to note the facilitator's performance in the evidence-based programming.
The facility's Standard Program Evaluation Protocol procedure states, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 29 youth in the
sample, 0% (0 of 29) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 7 youth
High = 19 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 66%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 19 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (29 of 29) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 66% (19 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Jacksonville Youth
Academy.
33 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 29 total youth sampled, only
28 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended weeks of service. Youth in the sample completed between 12
and 15 weeks of service, with an average of 13 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, all 29 included dosage in the EBS Module. Of those youth, 29 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 35 and 48 hours of
service, with an average of 40 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Impact of Crime is an intervention suitable for all youth. The Department recommends providing
this service to every youth in the program.
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups and by ensuring fidelity monitoring is
completed every month for each facilitator.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Jacksonville Youth Academy - TrueCore

program_status_note: Closed

service_type: Impact of Crime


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Male Healthy Relationships Bundle
SPEP Review Date(s): 3/9/2020
Florida Department of Juvenile Justice
Report Date(s): 3/9/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's documentation for the annual review period of March 2019 through February 2020
showed there were six staff trained to facilitate the Male Healthy Relationships bundle during the
review period. All six staff facilitated either the Teen Relationships group, Young Men’s Work, or
both during the review period.
Pursuant to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitators on the job
training plan. All facilitators will receive training prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Teen Relationship workbook and the Young Men's Work manual. The
Teen Relationships Workbook is broken down into six sections. The first section is evaluating your
relationship. Section two is understanding abuse. Section three is on social influences. Section
four is building healthy relationships. Section five is making good decisions. Section six is the
review. There are sixty-eight reproducible worksheets given to the youth during the sessions.
Each lesson has information on how the facilitator should present the material. There are no
separate student workbooks. The facilitators make copies of the current lessons to hand out to the
youth.
The Young Men's Work manual is broken down into twenty-six sessions. Each session has at
least one student exercise in them. There is a total of thirty-eight exercises during the group. The
manual provides information for the facilitator to conduct each session.
According to the program’s Gender-Specific SPEP Service “Bundle Proposal”, which was
approved for use by the Department, the Teen Relationships component assists youth to develop
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
healthy relationships and prevent dating abuse and domestic violence. Teen Relationships helps
teens identify healthy and unhealthy characteristics of relationships and to understand the
dynamics of emotional, physical, and sexual abuse in intimate relationships; youth explore and
clarify their own values and attitudes about gender roles and be empowered to make healthy
decisions about their own relationships.
In the same approved bundle proposal, it is specified Young Men’s Work is a gender-specific
component which teaches problem solving and violence prevention with gender-identity topics
ranging from power, violence, bullying, anger, and frustration, while also addressing healthy
relationships with women and interpersonal relationships between males. Additionally, Young
Men’s Work enhances social skills to cope with negative influences from others and the
environment that are a large part of the risk in youth transitioning back to disadvantaged
neighborhoods.
Young Men's Work and Teen Relationships are delivered one time each per week and are run as
open groups throughout the youth's length of stay. The youth will participate in the components
until a minimum of 40 combined contact hours are reached. The program runs the bundled
services on Thursdays and Fridays.
Pursuant the facility's Standard Program Evaluation Protocol (SPEP) procedure, each primary
service will be facilitated in accordance to its manual. The primary service will be conducted as
indicated in the manual/protocol; this includes adherence to the session plan, group size,
population, frequency, needed materials, duration, and interventions. The facilitator of the primary
service will specify the topic covered, interventions utilized, duration of the service, youth present,
and progress for each session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Male Healthy Relationships was unable to be observed. The program holds the Teen
Relationships group during times the reviewer was not on site.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of March 2019 through February 2020
shows the program had facilitator turnover. During the review period, all but one therapist and the
clinical director ended their employment at the program. The program has hired another therapist.
The turnover did not adversely affect groups. Groups were run according to the facility's schedule.
The groups had different trained facilitators conducting the groups during the annual review
period. The Male Healthy Relationships bundle (Teen Relationships and Young Men’s Work) is
run as an open group year-round.
Pursuant to the facility's operating procedure, Jacksonville Youth Academy utilizes a Probationary
Review Status Process designed for effective management of risk through individualized
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
coaching, training and verification activities of all new employees during their first 90 days. The
process encourages personal development and recognizes areas of improvement in an effort to
reduce staff turnover. In addition, the program provides positive incentives in an effort to improve
employee motivation and dedication and lead to better employee retention. Through this policy,
employees receive credit for years of service, employment referrals, and for contributing to the
positive culture, teamwork environment and for going above and beyond normal job expectations.
Each employee who receives credit for doing these things is entered in a monthly drawing at each
facility. In the event of facilitator turnover, the co-facilitator will assume the role of the facilitator for
the group. If a co-facilitator is not present, a new qualified facilitator for the primary service will
take over conducting the service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed showed fidelity monitoring took place in May and June 2019 only.
There was no other documentation of internal fidelity monitoring provided.
According to the facility's operating procedure, designated staff are responsible for conducting
fidelity monitoring of each facilitator of a primary service on a monthly basis. The staff conducting
the fidelity monitoring has received formal training by a qualified trainer in the primary service.
Fidelity monitoring is documented on a fidelity monitoring tool developed specifically for that
primary service by the developer of the manual, or an internal fidelity checklist created for
manualized or non-manualized curriculum delivered in the program. The staff performing the
fidelity monitoring of the primary service is verifying the facilitator is conducting the group in
accordance with the manual/protocol. This includes verifying the facilitator adheres to the session
plan, group size, population, frequency, needed materials, duration, and interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
According to the documentation, there were no corrective actions noted during the fidelity
monitoring.
Pursuant to the facility's Standard Program Evaluation Protocol procedure, based on the results of
the monthly fidelity monitoring, corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If, during additional fidelity
monitoring, the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event an issue identified during the primary service is detrimental to the youth’s
safety and/or treatment, the matter will be taken up directly with the program’s Human Resource
Manager and be addressed in accordance with the employee handbook.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
According to the documentation, the evaluation had an area to note the facilitator's performance
in this evidence-based programming.
The facility's Standard Program Evaluation Protocol procedure states, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 31 youth in the
sample, 0% (0 of 31) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 31 youth in the
sample, 87% (27 of 31) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 8 youth
High = 20 youth
Total Youth in Sample = 31 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 65%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 20 youth
Total Youth in Sample = 31 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (31 of 31) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 65% (20 of 31) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 73 95 77%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Jacksonville
Youth Academy.
33 youth completed the program during the review period. Of those youth, 31 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. There were no end dates entered
into the Juvenile Justice Information system (JJIS) Evidence-Based Services (EBS) module for
any youths. With no JJIS data to review, the SPEP score was reduced. The program must enter
dosage data into JJIS in order to receive any score in this area.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 31 total youth
sampled, only 27 included dosage in the EBS Module. Of those youth with correct dosage, 27
received at least the recommended hours of service. Youth in the sample completed between 41
and 93 hours of service, with an average of 75 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups; by ensuring fidelity monitoring is completed
every month for each facilitator; and by ensuring any corrective action needed is identified,
applied, and demonstrated.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Male Healthy Relationships


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Cannabis Youth Treatment
SPEP Review Period: March 15, 2020-March 14, 2021
SPEP Review Conducted: April 16, 2021
Florida Department of Juvenile Justice
Report Date:
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Cannabis Youth Treatment.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 5 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 25%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 3 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2020 through March 14,
2021 showed the program utilizes Cannabis Youth Treatment (CYT) as a primary service.
Cannabis Youth Treatment (CYT) training is an in-service training provided by the clinical director
or other approved person. The documentation shows there were three staff trained to facilitate
Cannabis Youth Treatment during the annual review period. The clinical director conducts the
monthly internal fidelity checks.
According to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitators on the job
training plan. All facilitators will receive training prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Cannabis Youth Treatment Series Volumes one and two. The manuals
provide step by step procedures for implementing the program. There are no student manuals.
The facilitators make copies of the worksheets and pass them out to the youth during group.
The program provides Cannabis Youth Treatment once a week on Wednesdays for one hour
each session. The program keeps all SPEP documents in a binder labeled Cannabis Youth
Treatment. The program states all youth who present with cannabis dependence are scheduled to
complete the primary service. The group is a closed group.
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Cannabis Youth Treatment was unable to be observed. The program is currently not
running the primary service group. The director of clinical services stated the program does not
have a licensed professional trained in the primary service.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period shows the program had facilitator
turnover which adversely affected the delivery of Cannabis Youth Treatment. The program had
two trained therapists and one clinical director who facilitated the primary service during the
annual review period. The clinical director and licensed therapist left the program. The non-
licensed therapist is the only facilitator left at the program. There were two cohorts of the primary
service delivered during the annual review period. The program listed the cohorts as group one
and group two. The dates of the cohorts were: group one 10/14/2020-12/16/2020 and group two
10/14/2020-12/16/2020.
Group one began with twelve youth and ended with nine youth. Group two began with eleven
youth and ended with nine youth. There were no other attendance sheets provided for the months
of April 2020 through September 2020 and January 2021 through March 2021. The director of
clinical services stated the program currently does not have a licensed therapist trained in the
primary service.
According to the facility's Standard Program Evaluation Protocol procedure, Jacksonville Youth
Academy utilizes a Probationary Review Status Process designed for effective management of
risk through individualized coaching, training and verification activities of all new employees
during their first 90 days. The process encourages personal development and recognizes areas of
improvement thus reducing staff turnover. In addition, the program provides positive incentives in
and effort to improve employee motivation and dedication and lead to better employee retention.
Through this policy, employees receive credit for years of service, employment referrals, and for
contributing to the positive culture, teamwork environment and for going above and beyond
normal job expectations. Each employee who receives credit for doing these things is entered in a
monthly drawing at each facility. In the event of facilitator turnover, the co-facilitator will assume
the role of the facilitator for the group. If a co-facilitator is not present, a new qualified facilitator for
the primary service will take over conducting the service.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period showed there was one fidelity
monitoring completed. The date of the fidelity monitoring was 10/21/2020. There was no other
documentation provided for the other months in the review period. The director of clinical services
stated there were no other internal fidelity checks found which were completed during the annual
review period. The clinical director conducts the monthly internal fidelity monitorings.
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no corrective
actions noted on the one fidelity monitoring which was completed. It is unknown if there were any
corrective actions needed due to the program only completing one internal monthly fidelity check
for the annual review period. Internal fidelity monitoring must occur regularly for points to be
awarded this section.
According to the facility's Standard Program Evaluation Protocol procedure, based on the results
of the monthly fidelity monitoring corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event that an issue identified during the primary service is detrimental to the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
youth’s safety and/or treatment, the matter will be taken up directly with the program’s Human
Resource Manager and be addressed in accordance with the employee handbook.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
According to the documentation, evaluations were completed on the three trained facilitators. The
evaluations had an area to note the facilitator's performance in the evidence-based programming;
however, there was not any specific information about the facilitator’s skill in delivering Cannabis
Youth Treatment. The evaluations only have a numerical value assigned along with listing the
primary service the staff is facilitating.
The facility's Standard Program Evaluation Protocol procedure states an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 12 weeks for this type of service. Of the 8 youth in the
sample, 50% (4 of 8) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 12 contact hours for this type of service. Of the 8 youth in the
sample, 100% (8 of 8) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 0 youth
Moderate-High = 3 youth
High = 4 youth
Total Youth in Sample = 8 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 4 youth
Total Youth in Sample = 8 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (7 of 8) youth scored Moderate to High-Risk to reoffend, for a score of
10 points.
Of the SPEP sample, 50% (4 of 8) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 5 20 25%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 77 100 77%
This SPEP report evaluates Cannabis Youth Treatment, an intervention delivered at Jacksonville
Youth Academy.
33 youth completed the program during the review period.
The program scored Low for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 12 weeks; the targeted number of
hours for this specific intervention is 12 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 4 points for Amount of Service: Duration. Of the 9 total youth sampled, all 8
included dosage with end dates in the EBS Module. Of those youth, 4 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 14 weeks of
service, with an average of 10 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 9 total youth
sampled, all 8 included dosage in the EBS Module. Of those youth, 8 received at least the
recommended hours of service. Youth in the sample completed between 12 and 15 hours of
service, with an average of 13 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring there are no unnecessary gaps in service, fidelity monitoring is completed every month
for each facilitator, any corrective action needed is identified, applied, and demonstrated, and
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Cannabis Youth Treatment


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Jacksonville Youth Academy
TrueCore Behavioral Solutions, LLC.
(Contract Provider)
4501 Lannie Road
Jacksonville, Florida 32218
Primary Service: Impact of Crime
SPEP Review Period: March 15, 2020-March 14, 2021
SPEP Review Conducted: April 16, 2021
Florida Department of Juvenile Justice
Report Date: 5/24/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of March 15, 2020 through March 14,
2021 showed the program utilizes Impact of Crime (IOC) as a primary service. IOC training is a
three-day training provided by a qualified trainer. The documentation shows there were three staff
trained to facilitate Impact of Crime during the annual review period. The trained staff are the
director of case management/transition manager and two case managers; one case manager
conducted all cohorts. The director of case management/transition manager conducts the internal
fidelity checks.
According to the facility's operating procedure, the facilitator for the primary service will have
training from a licensed clinician. This training will be documented in the facilitators on the job
training plan. All facilitators will receive training prior to conducting the service.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Impact of Crime: Addressing the Harm v 2.0. The manual opens with an
introduction to the Impact of Crime Curriculum. Then the manual goes over all seven chapters of
the group and provides a script for the facilitator to follow. The manual also has an appendix and
a student guide with all the student exercises. The youth are provided copies of the lesson for
each session.
The program provides Impact of Crime on Mondays and Wednesdays for one hour each session.
The program keeps all SPEP documents in a binder labeled Impact of Crime. The program states
all youth admitted to the program are scheduled to complete the primary service. The group is a
closed group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
According to the facility's Standard Program Evaluation Protocol procedure, each primary service
will be facilitated in accordance to its manual. The primary service will be conducted as indicated
in the manual/protocol; this includes adherence to the session plan, group size, population,
frequency, needed materials, duration, and interventions. The facilitator of the primary service will
specify the topic covered, interventions utilized, duration of the service, youth present, and
progress for each session on the sign-in sheet.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 4/7/2021 via Microsoft Teams. The group began at
5:30p. There were four youth present. The facilitator went over the group rules. The youth went
around the room and stated whether they had a good day or not and what their high and lows
were for the day. The facilitator discussed conflict. The facilitator asked the youth what they think
conflict is. The group then went over the words and definitions for chapter six. The group then
began chapter six exercise one. The youth were asked to decide if a statement was a myth or a
fact. The group ended at 6:28p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program did not have facilitator turnover. There was one
case manager who conducted all three cohorts during the annual review period. There were two
cohorts completed and one is currently ongoing. The dates of the cohorts are 4/20/2020-
7/20/2020, 9/14/2020-1/16/2021, and 2/8/2021-current.
The 4/20/2020 cohort started and ended with seven youth. The 9/14/2020 cohort began with ten
youth and ended with six youth.
The 2/8/2021 cohort began with ten youth and currently has four youth. The attendance sheets for
this cohort show the director of case management running additional sessions to allow the four
youth the ability to complete the primary service prior to being discharged from the program.
These additional sessions were provided on back-to-back days, but only had the four youth who
were close to program completion. The additional sessions were conducted on 3/1/2021,
3/2/2021, 3/3/2021, 3/4/2021, 3/5/2021, 3/11/2021, 3/12/2021, 3/16/2021, 3/18/2021, 3/19/2021,
and 3/20/2021. The director of case management started the additional sessions with the last
session and worked their way back to the chapter and objective where the rest of the youth were.
The documentation showed there was a gap in service of almost two months between 7/20/2020
and 9/14/2020.
According to the facility's Standard Program Evaluation Protocol procedure, Jacksonville Youth
Academy utilizes a Probationary Review Status Process designed for effective management of
risk through individualized coaching, training and verification activities of all new employees
during their first 90 days. The process encourages personal development and recognizes areas of
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
improvement thus reducing staff turnover. In addition, the program provides positive incentives, in
an effort to improve employee motivation and dedication and lead to better employee retention.
Through this policy, employees receive credit for years of service, employment referrals, and for
contributing to the positive culture, teamwork environment and for going above and beyond
normal job expectations. Each employee who receives credit for doing these things is entered in a
monthly drawing at each facility. In the event of facilitator turnover, the co-facilitator will assume
the role of the facilitator for the group. If a co-facilitator is not present, a new qualified facilitator for
the primary service will take over conducting the service.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period showed there was one case manager
who conducted all groups. The documentation showed there were eleven internal monthly fidelity
monitorings completed. The internal fidelity monitorings were completed on 4/27/2020, 5/11/2020,
6/10/2020, 7/13/2020, 9/16/2020, 10/1/2020, 11/10/2020, 12/30/2020, 1/9/2021, 2/17/2021, and
3/3/2021. One internal fidelity monitoring had the date 11/10/2020 on the first page, but the
signature page was dated for 10/14/2020. There were no other fidelity checks which were dated
for 10/14/2020. Each internal fidelity monitoring for the annual review period was conducted by
the director of case management/transition manager.
According to the facility's Standard Program Evaluation Protocol procedure, designated staff are
responsible for conducting fidelity monitoring of each facilitator of a primary service on a monthly
basis. The staff conducting the fidelity monitoring has received formal training by a qualified
trainer in the primary service. Fidelity monitoring is documented on a fidelity monitoring tool
developed specifically for that primary service by the developer of the manual, or an internal
fidelity checklist created for manualized or non-manualized curriculum delivered in the program.
The staff performing the fidelity monitoring of the primary service is verifying the facilitator is
conducting the group in accordance with the manual/protocol. This includes verifying the facilitator
adheres to the session plan, group size, population, frequency, needed materials, duration, and
interventions.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed showed there were no corrective actions noted during the fidelity
monitoring.
According to the facility's Standard Program Evaluation Protocol procedure, based on the results
of the monthly fidelity monitoring corrective action may be needed. The first stage of addressing
areas identified as needing improvement through fidelity monitoring is by providing coaching and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
supervision to the facilitator on that issue by a qualified facilitator in the primary service. The
additional coaching and/or supervision will be documented by the person conducting the fidelity
monitoring in a note and placed in the clinical supervision file. If during additional fidelity
monitoring the previously identified area addressed continues to be an issue, the next stage is
completing a written warning and placing the facilitator on a Performance Improvement Plan. The
Performance Improvement Plan will state the issue identified, the tasks the facilitator needs to
complete to correct the issue, and a target date for completion. The plan will then be reviewed
with the facilitator by the qualified facilitator conducting the monitoring, and the facilitator’s
supervisor. In the event that an issue identified during the primary service is detrimental to the
youth’s safety and/or treatment, the matter will be taken up directly with the program’s Human
Resource Manager and be addressed in accordance with the employee handbook.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period showed the program completed annual
evaluations for two of the trained facilitators. One of the provided evaluations did not list any of the
program’s primary services. The director of case management stated the facilitator was not
trained in Impact of Crime until December 2020 and had not conducted any groups before his
annual evaluation. The other annual evaluation received did indicate the staff member actively
facilitated Impact of Crime, but there were no specific remarks detailing the facilitator’s skill in
delivering the primary service. The director of case management conducted some of the primary
service groups, but there was no documentation of an annual evaluation. The director of case
management stated “I have requested a copy of mine from Human Resources which is at another
location; however, I haven't received it yet.”
The facility's Standard Program Evaluation Protocol procedure states, an annual evaluation of
performance will be conducted on all the facilitators of each primary service. The performance
evaluation is comprised of feedback from the fidelity monitoring tools as well as an assessment of
the facilitator’s skill in delivering the primary service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 22 youth in the
sample, 50% (11 of 22) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 22 youth in the
sample, 86% (19 of 22) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 8 youth
High = 9 youth
Total Youth in Sample = 22 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 22 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (20 of 22) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 41% (9 of 22) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Jacksonville Youth
Academy.
33 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 22 total youth sampled, all
22 included dosage with end dates in the EBS Module. Of those youth, 11 received at least the
recommended weeks of service. Youth in the sample completed between 14 and 33 weeks of
service, with an average of 18 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 22 total youth
sampled, all 22 included dosage in the EBS Module. Of those youth, 19 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 1 and 36 hours of
service, with an average of 31 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Impact of Crime is an intervention suitable for all youth. The Department recommends providing
this service to every youth in the program.
Jacksonville Youth Academy can optimize their SPEP Quality of Service Delivery score by
ensuring there are no unnecessary gaps in service and each facilitator’s performance evaluation
includes discussion of their skill in delivering this primary service.
Jacksonville Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10