summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Anger Management for Substance Abuse and Mental
Health Clients
SPEP Review Date(s): 5/22/2020
Florida Department of Juvenile Justice
Report Date(s): 5/22/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Substance Abuse and Mental
Health Clients.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 20%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Therapy staff are trained to provide the Anger Management for Substance Abuse and Mental
Health Clients curriculum. The program provided training records for review. There were three
staff trained during the review period. There were a total of ten therapists who have been trained
in the Anger Management for Substance Abuse and Mental Health Clients curriculum during the
review period.
The therapists are trained by the designated mental health clinician authority (DMHCA) in an
informal training. The training is documented on sign in sheets maintained by the DMHCA.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Anger Management for Substance Abuse and Mental Health Clients is held on Tuesdays and
Thursdays from 2:15pm to 3:15pm and 3:15pm to 4:15pm. The attendance sheets for the review
period were provided. The attendance sheets are kept in the designated mental health clinician
authority's (DMHCA) office, rubber-banded together by month with all other mental health groups.
This group is held as an open group, meaning youth can enter the group upon admission to the
program.
Youth are selected to receive this curriculum based upon the results of the Residential
Assessment for Youth (RAY). It can be provided for mental health and/or substance abuse clients,
but a diagnosis is not required, according to the facilitator manual. The requirements to facilitate
this curriculum are facilitators should be trained in cognitive behavioral therapy, group therapy,
and substance abuse treatment.
The DMHCA provided the manual for review. It is maintained in a binder. There are twelve
chapters to complete. The instructor's manual provides the dosage infromation of twelve-weeks
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
and ninety minute sessions, but can be held as sixty minutes. The sessions provide a script and
outline of how the session should be facilitated. The student guide provides all worksheets for the
lessons. The youth do not receive individual workbooks. The facilitators make copies of the
worksheets for the youth to complete during group.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Anger Management for Substance Abuse and Mental Health Clients was not
observed during this review period. Group observations will resume during quarterly reviews
scheduled throughout the annual review period. This indicator is rated as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
All therapists are trained in the Anger Management for Substance Abuse and Mental Health
Clients curriculum. Should the program experience turnover in therapy staff, another therapist
would continue to facilitate the group. Three therapists are no longer at the program; however this
did not interfere with group facilitation. There have been no gaps in service and no turnover during
this review period.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The therapists are trained conduct internal fidelity monitoring for each other. From a review of the
internal fidelity monitoring forms, the more seasoned therapists conduct the observations on the
newly trained therapists. There have been a total of ten therapists who have facilitated Anger
Management for Substance Abuse and Mental Health Clients during the review period. Not all the
therapists facilitated monthly, with the exception of three. All internal fidelity monitoring forms were
provided for review. There were no internal fidelity monitoring forms missing for the review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There were no significant issures or corrective actions noted on the internal fidelity monitoring
forms for Anger Management for Substance Abuse and Mental Health Clients. Any suggestions or
comments are documented on the internal fidelity monitoring forms.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Three therapists are still within their first year and have not received their annual performance
evaluation. Four therapists have received their annual performance evaluation. There is a section
on the performance evaluations which assesses for group facilitation and training.
Three therapists left during the review period and did not receive a performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 24 weeks for this type of service. Of the 91 youth in the
sample, 51% (46 of 91) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 91 youth in the
sample, 0% (0 of 91) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 11 youth
Moderate-High = 34 youth
High = 38 youth
Total Youth in Sample = 91 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 38 youth
Total Youth in Sample = 91 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (83 of 91) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (38 of 91) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 77 95 81%
This SPEP report evaluates Anger Management for Substance Abuse and Mental Health Clients,
an intervention delivered at Okaloosa Youth Academy.
91 youth completed the program during the review period. Of those youth, all 91 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 91 total youth sampled, only
89 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 46
received at least the recommended weeks of service. Youth in the sample completed between 19
and 51 weeks of service, with an average of 26 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 91 total youth
sampled, only 89 included dosage in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended hours of service. Youth in the sample completed between 12
and 24 hours of service, with an average of 15 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Okaloosa Youth Academy

summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Date(s): 05/22/2020
Florida Department of Juvenile Justice
Report Date(s): 05/22/2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 20%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Case management staff are formally trained to facilitate the Impact of Crime (IOC) curriculum.
There are four case management staff and the case management supervisor trained to facilitate
the IOC curriclulum. All training documentation has been provided for review. The certificates of
completion were located in the Standardized Program Evaluation Protocol (SPEP) binder or in the
Department's learning management system, SkillPro.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
Impact of Crime (IOC) groups are currently being held on Mondays and Wednesdays from
4:00pm - 5:00pm. The group schedule has changed some over the review period. The case
management supervisor has worked diligently to ensure all youth are able to attend IOC groups.
The program maintains the current manual for IOC. Each faciltiator has their own manual they use
during groups. The youth are provided the worksheets from the youth workbooks during group
time.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Impact of Crime did not take place during this review. Group observations will
take place during quarterly reviews throughout the annual review period. This indicator is rated as
non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There has not been turnover in case management staff during this review period. All case
management staff has been formally trained in the Impact of Crime curriculum. Should there be
any turnover, another case manager or the case management supervisor would complete the
group to avoid any gaps in service. There were no significant gaps in service during this review
period.
Eight cohorts were completed during the review period; two additional cohorts were ongoing at
the time of this review. The first cohort started June 24, 2019 and completed on October 7,
2019.This group was held on Mondays and Wednesdays from 2:15pm to 3:15pm. The group
started with eight youth and was completed with seven youth. One youth left between September
4, 2019 and September 30, 2019; however, it was not documented on the attendance forms.
There was a gap between September 4, 2019 and September 30, 2019 and no explanation was
documented.
The second cohort started May 7, 2019 and completed on August 15, 2019 with nine youth
starting the group and ending with 8 youth. The group was held on Tuesdays and Thursdays from
2:15pm-3:15pm.The group started with eight youth. Per the case management supervisor, three
youth are receiving an extra session outside of the group due to needing to complete the group
prior to being released from the program. This reviewer discussed with the case management
supervisor this is an extenuating circumstance with trying to get the youth through IOC prior to
release. Individual attendance sheets were completed for the youth completing individual
sessions.
The third cohort began July 30, 2019 and completed November 12, 2019 with two new facilitators.
The group was held Tuesdays and Thursdays from 4:00pm to 5:00pm and had fifteen youth in the
group,a large number of youth for this curriculum. It was suggested a seasoned faciltiator sit in the
group with the two new facilitators to assist when needed.
The fourth cohort began September 3, 2019 and completed on December 12, 2019. The group
began and ended with eight youth. The fifth cohort began October 21, 2019 and completed
February 4, 2020. The group started with eight youth and ended with seven youth. One youth left
during the group and did not successfully complete the group.
The sixth cohort began October 21, 2019 and completed on February 4, 2020. This group began
with eight youth and ended with three youth. This reviewer discussed this with the case
management supervisor. During this group, youth were transferred from a different program in the
middle of an IOC group at their program. The case management supervisor continued the IOC
group with the transferred youth and the youth from her group went into another IOC group to
complete it. All youth were able to successfully complete IOC prior to finshing the program.
The seventh cohort began December 24, 2019 and completed on March 12, 2020. The group
began and ended with eight youth. The eight cohort began January 13, 2020 and ended April 7,
2020. The group began with eight youth and ended with seven youth. One youth was transferred
during the group and this was documented on the attendance sheets.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
There are two cohorts currently running at the time of this review. One began on February 3, 2020
and the second on March 26, 2020. Both groups have eight youth in attendance. All cohorts met
with the SPEP dosage requirements per a review of the attendance forms.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring of Impact of Crime is conducted by the case management supervisor
the majority of the time. Other seasoned facilitators assist in conducting internal fidelity monitoring
when needed. All internal fidelity monitorng forms were provided for review. There were no
missing internal fidelity monitoring forms during this review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The case management supervisor conducts the majority of internal fidelity monitoring for Impact
of Crime (IOC) groups. The case management supervisor addresses any issues with a facilitator
at the time of their debriefing after group. The two senior case managers rotate conducting
internal fidelity monitoring for the case management supervisor when needed. Suggestions were
made on most of the internal fidelity monitoring forms and any discussion with the facilitators were
also documented. There were no significant issues or corrective actions noted on the internal
fidelity monitoring forms.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The two facilitators for Impact of Crime received an annual performance evaluation during the
review period. There is not a section on the annual performance evaluations to evaluate group
facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 88 youth in the
sample, 15% (13 of 88) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 88 youth in the
sample, 49% (43 of 88) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 11 youth
Moderate-High = 32 youth
High = 37 youth
Total Youth in Sample = 88 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 42%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 37 youth
Total Youth in Sample = 88 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (80 of 88) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 42% (37 of 88) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 57 85 67%
This SPEP report evaluates Impact of Crime, an intervention delivered at Okaloosa Youth
Academy.
91 youth completed the program during the review period. Of those youth, 88 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 88 total youth sampled, only
51 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 6
and 17 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 88 total youth
sampled, only 63 included dosage in the EBS Module. Of those youth with correct dosage, 43
received at least the recommended hours of service. Youth in the sample completed between 5
and 33 hours of service, with an average of 23 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service, and by ensuring each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Okaloosa Youth Academy

program_status_note: Open

service_type: Impact of Crime


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Seven Challenges
SPEP Review Period: May 1, 2020 - April 30, 2021
SPEP Review Conducted: June 14, 2021
Florida Department of Juvenile Justice
Report Date: 8/9/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seven Challenges as a primary service for the substance abuse youth
admitted to the program. The program's contract requires therapy staff to be trained to facilitate
the Seven Challenges curriculum.
According to the clinical director, all therapists are trained in the Seven Challenges curriculum. A
review of documentation confirmed there were six therapists and the clinical director currently
trained in the Seven Challenges curriculum. The therapists rotate conducting Seven Challenges
on a regular basis. All six therapists facilitated the group during the twelve month review period.
The clinical director conducts an informal training for the therapists. The therapists do receive a
certificate of completion. All training documentation was provided for review.
At the time of this review there are two main facilitators for the Seven Challenges curriculum who
work with the youth in the substance abuse program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Seven Challenges manual for review. The youth have a separate
workbook for this curriculum they work on throughout their time in the group. Facilitators use the
guide to read through scenarios and utilize the discussion questions with the group, applying the
scenarios to real-life events and experiences. There is no specific script for the curriculum.
The sign-in sheets for the Seven Challenges groups are maintained in the clinical director's office
rubber-banded together by month. Internal fidelity monitoring forms are kept in a binder in the
clinical director's office. All documentation was provided for this review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
The Seven Challenges group is held Monday through Friday from 2:15pm - 3:15pm and 3:15pm -
4:15pm. All youth with a substance abuse diagnosis are eligible to attend Seven Challenges.
Seven Challenges is held as an open group and run throughout the year.
The licensed therapist (in for the clinical director on the day of review) confirmed the youth
received combined services of Seven Challenges and Anger Management for Substance Abuse
and Mental Health clients on Tuesdays and Thursdays each week.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges was not able to be observed during this review period. The
reviewer attempted to observe groups virtually due to COVID protocols; however, the program
was unable to conduct the groups online due to internet issues.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A discussion with the clinical director confirmed there has been no turnover in the therapy staff at
the program during the twelve month review period. The therapists rotate facilitating Seven
Challenges groups on a regular basis. Should a therapist be out or leave employment another
therapist would continue with the group. Having the therapists rotate facilitation creates minimal
impact to the youth if a facilitator leaves.
All sign-in sheets for the twelve month review period were made available for review. There were
no gaps in service for the twelve month review period. All groups were held for one hour each
session.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for other trained therapists to monitor each other facilitating the Seven
Challenges curriculum. All therapists conducted Seven Challenges groups during the twelve-
month review period.
Fifty-six out of fifty-six internal fidelity monitoring forms were provided and reviewed. There were
no missing internal fidelity monitoring forms for the twelve month review period. The fidelity
monitoring forms include a section for feedback for the facilitators. All facilitators did receive
feedback on the internal fidelity monitoring forms.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the clinical director to address any significant issues or corrective
actions with the facilitators of Seven Challenges. A review of all internal fidelity monitoring forms
provided documentation that there were no corrective actions for the group faciltiators during the
twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations include a section for group facilitation. All facilitators for the
Seven Challenges curriculum received an annual performance evaluation. This also included
feedback on facilitation performance and a rating score.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 24 weeks for this type of service. Of the 33 youth in the
sample, 45% (15 of 33) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 33 youth in the
sample, 100% (33 of 33) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 12 youth
High = 16 youth
Total Youth in Sample = 33 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 16 youth
Total Youth in Sample = 33 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (30 of 33) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 48% (16 of 33) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 87 95 92%
This SPEP report evaluates Seven Challenges, an intervention delivered at Okaloosa Youth
Academy.
59 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 33 total youth sampled, all
33 included dosage with end dates in the EBS Module. Of those youth, 15 received at least the
recommended weeks of service. Youth in the sample completed between 16 and 46 weeks of
service, with an average of 26 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 33 total youth
sampled, all 33 included dosage in the EBS Module. Of those youth, 33 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 71 and 170 hours of
service, with an average of 106 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Seven Challenges


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Anger Management for Substance Abuse and Mental
Health Clients
SPEP Review Period: May 1, 2020 - April 30, 2021
SPEP Review Conducted: June 14, 2021
Florida Department of Juvenile Justice
Report Date: 8/9/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Substance Abuse and Mental
Health Clients.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Anger Management for Substance Abuse and Mental Health Clients (Anger
Management) as a primary service. The therapy staff are trained to facilitate Anger Management
groups, in accordance with the program's contract. The clinical director and five therapists were
trained to facilitate Anger Mangement groups.
The clinical director conducts informal training with the therapy staff. The clinical director
maintains training sign-in sheets for documentation of training. All training documentation was
provided for the twelve month review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual for Anger Management for Substance Abuse and Mental Health Clients is maintained
in a binder. There are twelve chapters to complete. The instructor's manual provides the dosage
information of twelve weeks and ninety minute sessions, but can be held for sixty minutes. The
sessions provide a script and outline of how the session should be facilitated. The student guide
provides all worksheets for the lessons. The youth do not receive individual workbooks. The
facilitators make copies of the worksheets for the youth to complete during group.
Anger Management for Substance Abuse and Mental Health Clients (Anger Management) is held
on Tuesdays and Thursdays from 2:15pm to 3:15pm and 3:15pm to 4:15pm for all youth in the
program, according to a review of the program's group schedule.
According to a discussion with the licensed therapist in for the clinical director on the day of
review, the substance abuse youth received combined services of Seven Challenges and Anger
Management on Tuesdays and Thursdays each week. The attendance sheets were provided for
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
review for the twelve month review period. The attendance sheets are kept in the clinical director's
office, rubber-banded together by month with all other mental health groups.
A review of the program's contract confirms all youth are eligible to attend this group. Anger
Management is held as an open group, meaning youth can enter the group upon admission to the
program.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Anger Management for Substance Abuse and Mental Health Clients was not able to
be observed during this review period. The reviewer attempted to observe groups virtually due to
COVID protocols; however, the program was unable to conduct the groups online due to internet
issues.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Anger Management for Substance Abuse and Mental Health Clients is an open group and is held
on Tuesdays and Thursdays. There are two groups held for mental health youth and two groups
held for substance abuse youth to ensure all youth are able to attend. All sign-in sheets were
provided for this review period. There were no gaps in service during this review period.
The program has not experienced turnover during this review period. All therapists are trained in
the curriculum and rotate facilitation of the group. This rotation allows the youth to become familiar
with all therapy staff. In the event of turnover, the therapists would continue with their rotation.
This limits the disruption the youth for the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy is for other trained therapists to conduct internal fidelity monitoring for the
facilitators of Anger Management for Substance Abuse and Mental Health Clients (Anger
Management). During the last quarter of review there were three main facilitators for Anger
Management groups. However, all therapists have rotated conducting Anger Management groups
throughout the twelve month review period.
The internal fidelity monitoring forms are maintained in a binder and kept in the clinical director's
office. All internal fidelity monitoring forms were provided for the twelve month review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Seventy-eight out of seventy-eight internal fidelity monitoring forms were reviewed. There were no
missing internal fidelity monitoring forms.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the clincal director to address any issues or corrective actions with the
facilitators of Anger Management for Substance Abuse and Mental Health Clients.
All internal fidelity monitoring forms were reviewed for the twelve-month review period. There is a
section for feedback, corrective actions, and issues to be documented. The internal fidelity
monitoring forms had feedback for each facilitator each month. There were no corrective actions
noted for the twelve month review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual performance evaluations for the facilitators of Anger Management
for Substance Abuse and Mental Health Clients (Anger Management). The annual performance
evaluations include a specific section for group facilitation and includes the specific groups the
employee facilitates.
All facilitators for Anger Management had received an annual performance evaluation during the
twelve month review period. There was no specific evaluation of each facilitator's skill in delivering
the intervention.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 24 weeks for this type of service. Of the 59 youth in the
sample, 54% (32 of 59) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 59 youth in the
sample, 0% (0 of 59) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 22 youth
High = 24 youth
Total Youth in Sample = 59 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 24 youth
Total Youth in Sample = 59 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 54% (32 of 59) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 41% (24 of 59) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 77 95 81%
This SPEP report evaluates Anger Management for Substance Abuse and Mental Health Clients,
an intervention delivered at Okaloosa Youth Academy.
59 youth completed the program during the review period. Of those youth, all 59 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 59 total youth sampled, only
57 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 32
received at least the recommended weeks of service. Youth in the sample completed between 12
and 49 weeks of service, with an average of 27 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 59 total youth
sampled, all 59 included dosage in the EBS Module. Of those youth, 0 received at least the
recommended hours of service. Youth in the sample completed between 20 and 24 hours of
service, with an average of 24 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Period: May 1, 2020 - April 30, 2021
SPEP Review Conducted: June 14, 2021
Florida Department of Juvenile Justice
Report Date: 8/9/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program's contract documents case management staff is trained to conduct Impact of Crime
(IOC) groups at the program. There are five total staff trained to facilitate Impact of Crime: four
case managers and the case management supervisor. All staff attended a formal training and all
documentation was provided for review. The case management staff rotate faciliting Impact of
Crime cohorts throughout the twelve month review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the current IOC manual. The manual includes a detailed script for the
facilitator, all lesson worksheets, and the student workbook. The student workbook can be copied
and provided to the youth during each cohort.
All records were provided for review for Impact of Crime (IOC) groups. IOC is held as closed
groups, meaning once a group has started youth cannot enter a group and must wait for the next
groups to begin. Groups are held Mondays through Thursdays from 4:00pm to 5:00pm.
All sign-in sheets, internal fidelity monitoring forms, and training records are maintained in a
binder in the case management supervisors office. All youth are eligible to attend this curriculum.
There are no specific case managers assigned to facilitate IOC for the youth in the program. All
youth from the programs are put together in groups for IOC. According to the case management
supervisor, the case managers rotate based upon where the need is for the group in the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was not observed during this review period and will be rated as non-
applicable. This reviewer did set up an observation; however, no one from the program called in
for the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The case management supervisor and a review of the program's documentation showed the
program has not experienced any turnover during the twelve month review period. If the program
experiences turnover during a cohort, another trained case manager or the case management
supervisor will continue the group.
The program has completed ten cohorts of Impact of Crime during the twelve month review
period.
The first cohort was started on March 24, 2021 and ended on June 25, 2020. Seven youth started
the group and six youth ended the group successfully.
The second cohort was started on March 24, 2020 and ended on July 9, 2020. Eight youth started
the group and seven youth successfully completed the group.
The third cohort was started on March 26, 2020 and ended on June 30, 2020. Eight youth started
the group and eight youth successfully completed the group.
The fourth cohort was started on June 1, 2020 and ended on September 14, 2020. The group
started with five youth and ended successfully with five youth.
The fifth cohort was started August 10, 2020 and ended on September 21, 2020. This reviewer
spoke with the case management supervisor regarding the short time for the group. Per the case
management supervisor, the groups were held for two hours at a time. There were only three
youth in the group and these youth needed to receive the service prior to their release. It was
decided the groups would be held for two hours at a time so the youth could complete the
curriculum prior to release. Three youth started the cohort and three youth successfully completed
the cohort.
The sixth cohort began on September 15, 2020 and ended on December 17, 2020. This group
started with eight youth and eight youth successfully completed. The weeks were short for this
group due to some of the groups being held for an hour and a half or two hours at a time. When
asked about this, it was, again, to ensure all youth in the group were able to complete the
curriculum prior to their release.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The seventh cohort began on November 17, 2020 and was ended on February 23, 2021. Eight
youth started this group and eight you successfully completed. Four of these sessions were held
for two hours at a time.
The eighth cohort began on December 8, 2020 and ended on March 25, 2021. Eight youth started
the group and seven youth successfully completed. One youth was transferred from the program
during the cohort.
The ninth cohort began on December 28, 2020 and ended on April 7, 2021. Eight youth started
the group and eight youth successfully completed.
The tenth cohort began on January 19, 2021 and ended on May 6, 2021. Eight youth started this
cohort and eight youth successfully completed.
Three cohorts are ongiong at the time of this review. One cohort started on March 16, 2021, with
eight youth attending. A second cohort was started April 6, 2021, with eight youth attending.
A third cohort was started on May 17, 2021, with one youth doing individual sessions. According
to the case management supervisor and case manager, the youth was transferred in from another
program where he had started the Impact of Crime curriculum. Since groups had already been
started when he was transferred into the program, it was decided to do individual sessions in
order to allow him to successfully complete the group.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The case management supervisor conducts all internal fidelity monitoring for the Impact of Crime
curriculum. All case managers rotate conducting the Impact of Crime groups.
Fourty-five out of fourty-five internal fidelity monitoring forms were provided for the twelve month
review period. There were no missing internal fidelity monitoring forms for the twelve month
review period.
All Impact of Crime facilitators were included in the internal fidelity monitoring. The program is
using the correct internal fidelity monitoring forms and the forms include a feedback section for the
facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions for Impact of Crime stemming
from internal fidelity monitoring. The case management supervisor addresses any issues or
corrective actions with the case management supervisor. All internal fidelity monitoring forms
were provided for review and there were no corrective actions noted.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
All case management yearly performance evaluations were provided for review. There is a
section for delinquency intervention service facilitation. This section lists all services the facilitator
is trained in. There was no specific evaluation of each facilitator's skill in delivering the
intervention.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 58 youth in the
sample, 26% (15 of 58) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 58 youth in the
sample, 90% (52 of 58) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 8 youth
Moderate-High = 22 youth
High = 24 youth
Total Youth in Sample = 58 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 41%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 24 youth
Total Youth in Sample = 58 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (57 of 58) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 41% (24 of 58) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Okaloosa Youth
Academy.
59 youth completed the program during the review period. Of those youth, 58 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 58 total youth sampled, only
51 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 15
received at least the recommended weeks of service. Youth in the sample completed between 7
and 17 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 58 total youth
sampled, all 58 included dosage in the EBS Module. Of those youth, 52 received at least the
recommended hours of service. Youth in the sample completed between 19 and 52 hours of
service, with an average of 26 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Seven Challenges
SPEP Review Period: May 1, 2021 - April 30, 2022
SPEP Review Conducted: May 2, 2022 - May 6, 2022
Florida Department of Juvenile Justice
Report Date: 8/10/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes the Seven Challenges curriculum as a primary service. Seven Challenges is
designed to motivate a decision and commitment to change, as well as support success in
implementing the desired change. Seven Challenges helps juveniles address their drug problems
and co-occurring life skill deficits, situational problems, and psychological problems. Seven
specific challenges provide a framework for helping youth think through decisions about their lives
and their use of alcohol and other drugs.
The program trains therapy staff to conduct Seven Challenges groups in the program. There have
been seven staff trained over the annual review year. There have been three main therapists for
the B Dorm, which houses the youth committed to the substance abuse program at Okaloosa
Youth Academy. All training records were provided for review.
The previous clinical director conducted trainings with the staff as they were hired. The training
consisted of the clinical director going through lessons in the manual with the therapy staff. The
therapy staff receive a certificate of completion once they have successfully completed the
training.
The trained therapists conduct monthly internal fidelity monitoring on each other.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the current Seven Challenges manual and student journals. The manual
contains all lessons, worksheets, and a generalized script for the facilitators. The youth have a
separate workbook for this curriculum they work in throughout their time in the group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Seven Challenges groups are held Mondays through Fridays from 2:15pm to 3:15pm and 3:15pm
to 4:15pm. All sign-in sheets are maintained in a binder kept in the clinical director's office.All
youth housed in the substance abuse program within Okaloosa Youth Academy are eligible to
receive the Seven Challenges curriculum. The groups are held as open groups, meaning the
youth may enter the group upon admission to the program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A Seven Challenges group was observed on Wednesday, September 8, 2021. There were five
out of seven youth present. There was one staff member in group. The staff member was friendly
and approachable. It appeared the youth had a healthy respect for this staff and the therapist. The
facilitator has been conducting the Seven Challenges curriculum for a few years.
There was one new youth in the group. The facilitator welcomed the new youth and reviewed the
group expectations and how the curriculum is run. The facilitator conducted a group check-in with
all the youth. Two youth shared in the reading for this group. The topic of the reading was "Raging
Anger". The group discussed this topic openly and all youth were engaged and participated in the
discussion. Today was a journaling day. For the youth who had completed all seven challenges,
they worked on completing their prevention plan for their release. The new youth was unable to
receive a journal due to the journals not being delivered when expected. The facilitator sat with
this youth and discussed the curriculum and group in more detail.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process in the event of turnover in facilitators is to have another trained therapist
conduct the Seven Challenges group. The therapists rotate facilitation of groups so the youth are
familiar with the therapists.
The program experienced turnover duing the annual review period. The clinical director and three
therapists left during the annual review period. Seven Challenges groups are held as open
groups, therefore the staff left during cohorts. There were other trained facilitators available to
continue facilitating the Seven Challenges groups.
All sign-in sheets were provided for review. There were no gaps in service during the annual
review period. There were between eight and ten (10) youth in all of the Seven Challenges groups
throughout the annual review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process is for another trained therapist to conduct monthly internal fidelity
monitoring for each other. Thirty-eight (38) out of thrity-nine (39) monthly internal fidelity
monitorings were conducted for facilitators of Seven Challenges, based upon when therapists
facilitated the group. One facilitator did not receive internal fidelity monitoring for February 2022.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process is for the clinical director to address any significant issues or corrective
actions with the therapists for Seven Challenges.
The internal fidelity monitoring forms include a section where any issues can be documented. A
review of internal fidelity forms for the annual review period confirmed there were no significant
issues or corrective actions documented.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluation for the program does not include a section to address group
facilitation. It was suggested to the clinical director for an amendment to be added to the
performance evaluation to include all services the therapists facilitate, including Seven
Challenges.
Four therapists did receive annual performance evaluations during the annual review period. The
previous clinical director and three therapists left employment before receiving their annual
performance evaluations.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 34 youth in the
sample, 26% (9 of 34) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 34 youth in the
sample, 100% (34 of 34) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 9 youth
High = 18 youth
Total Youth in Sample = 34 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 18 youth
Total Youth in Sample = 34 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (31 of 34) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 53% (18 of 34) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 85 95 89%
This SPEP report evaluates Seven Challenges, an intervention delivered at Okaloosa Youth
Academy.
75 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 34 total youth sampled, only
33 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 9
received at least the recommended weeks of service. Youth in the sample 75 between 19 and 33
weeks of service, with an average of 23 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 34 total youth
sampled, all 34 included dosage in the EBS Module. Of those youth, 34 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 80 and 169 hours of
service, with an average of 103 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' 75 recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 OYA 7C.pdf