summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Anger Management for Substance Abuse and Mental
Health Clients
SPEP Review Period: May 1, 2021 - April 30, 2022
SPEP Review Conducted: May 2, 2022 - May 6, 2022
Florida Department of Juvenile Justice
Report Date: 8/10/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Anger Management for Substance Abuse and Mental
Health Clients.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Anger Management for Substance Abuse and Mental Health Clients (Anger
Management). The Standardized Program Evaluation Protocol (SPEP) only reviews for the
mental health youth in the program who receive this service. Anger Management for Substance
Abuse and Mental Health Clients is a cognitive-behavioral therapy.
All therapy staff are trained to conduct Anger Management groups in the program. There have
been five trained therapists who have facilitated the Anger Management groups during the twelve
(12)-month review period. There are three main therapists for the curriculum.
The therapy staff monitors each other for internal fidelity monitoring on a monthly basis.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual for Anger Management for Substance Abuse and Mental Health Clients (Anger
Mangement) is maintained in a binder in the clinical director's office. There are twelve (12)
chapters to complete. The instructor's manual provides the dosage information of twelve (12)
weeks and ninety (90) minute sessions, but can be held for sixty (60) minutes. The sessions
provide a script and outline of how the session should be facilitated. The student guide provides
all worksheets for the lessons. The youth do not receive individual workbooks. The facilitators
make copies of the worksheets for the youth to complete during group.
Anger Management for Substance Abuse and Mental Health Clients is held on Tuesdays and
Thursdays from 2:15pm to 3:15pm and 3:15pm to 4:15pm. All documentation for the Anger
Management groups is kept in a SPEP binder maintained in the clinical director's office. The
binder includes all sign-in sheets for the groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
All youth who are receiving mental health overlay services are able to receive the Anger
Management group. These groups are held as open, meaning the youth can join the group upon
admission to the program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Anger Management for Substance Abuse and Mental Health Clients (Anger
Management) was held on Wednesday, May 4, 2022 from 2:15pm to 3:15pm. There were six
youth, one facilitator, and one direct care staff present. The youth were reviewing the anger
management cycle. The faciltiator is comfortable conducting this group as she facilitates it on a
consistent basis. This reviewer had a copy of the lesson to follow. The facilitator engaged the
youth and led productive conversations regarding the topic.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process in the event of facilitator turnover is to have another therapist continue
facilitating the groups. The program has the therapists rotate facilitation of Anger Management for
Substance Abuse and Mental Health Clients so the youth are familiar with all facilitators.
The program did experience turnover during the twelve (12)-month review period. Three
therapists and the clinical director left employment. The groups are held as open, therefore the
therapists and clinical director left while groups were being held.
All sign-in sheets were provided for review. There were no gaps in service during the annual
review period. There were between eight and ten (10) youth in all of the groups.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process for interal fidelity monitoring of Anger Management for Substance Abuse
and Mental Health Clients is for the trained therapists to observe one another on a monthly basis.
There were forty-two (42) out of forty-five (45) internal fidelity monitoring observations conducted
during the annual review period.
One therapist was missing fidelity monitoring for October 2021, one therapist was missing fidelity
for January 2022, and one therapist was missing fidelity monitoring for September 2021.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any significant issues or corrective actions with the facilitators for
Anger Management for Substance Abuse and Mental Health Clients. The internal fidelity
monitoring forms include a section to document any significant issues or corrective actions.
All internal fidelity monitoring forms were provided for review. There were no significant issues or
corrective actions documented during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluation for the program does not include a section to address group
facilitation. It was suggested to the clinical director for an amendment to be added to the
performance evaluation to include all services the therapists facilitate.
Four therapists did receive annual performance evaluations during the annual review period. The
previous clinical director and three therapists left employment before receiving their annual
performance evaluations.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 75 youth in the
sample, 23% (17 of 75) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 75 youth in the
sample, 0% (0 of 75) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 21 youth
High = 37 youth
Total Youth in Sample = 75 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 37 youth
Total Youth in Sample = 75 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (67 of 75) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 49% (37 of 75) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 75 95 79%
This SPEP report evaluates Anger Management for Substance Abuse and Mental Health Clients,
an intervention delivered at Okaloosa Youth Academy.
75 youth completed the program during the review period. Of those youth, 75 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 75 total youth sampled, only
73 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 17
received at least the recommended weeks of service. Youth in the sample completed between 10
and 39 weeks of service, with an average of 20 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 75 total youth
sampled, only 74 included dosage in the EBS Module. Of those youth with correct dosage, 0
received at least the recommended hours of service. Youth in the sample completed between 16
and 33 hours of service, with an average of 24 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 OYA AMSAMHC.pdf


state: FL

facility_name: Okaloosa Youth Academy

summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Period: May 1, 2021 - April 30, 2022
SPEP Review Conducted: May 2, 2022 - May 6, 2022
Florida Department of Juvenile Justice
Report Date: 8/11/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. The Impact of Crime curriculum
was created by the Florida Department of Juvenile Justice Office of Residential Services to assist
residential programs with following a balanced and restorative justice approach.
Case management staff are trained to conduct IOC groups in the program. Seven trained staff
facilitated IOC groups during the annual review year. Two of the seven staff were trained during
the annual review period and those training records were provided for review.
The case management supervisor conducts internal fidelity monitoring. If the case management
supervisor is facilitating a group, a senior case manager will conduct internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Impact of Crime (IOC) manual. The manual in use is not the most
recent version available. This reviewer requested four new IOC manuals be sent to the program
from residential services at the Department of Juvenile Justice.
Impact of Crime groups are held on Mondays and Wednesdays or Tuesdays and Thursdays in
the afternoons.
The program maintains the documentation for IOC groups in a binder kept in the case
management supervisor's office. The binder contains sign-in sheets, internal fidelity monitoring
documentation, and training documentation. All youth in the program are eligible to attend IOC
groups. Impact of Crime groups are closed groups, meaning no youth can be added to the group
once the initial lesson has been completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime (IOC) was observed on November 15, 2021. There were eight
youth, one facilitator, and one youth care worker present for the group. The group lasted from
2:45pm to 3:45pm. The group time was moved up on this date due to therapy groups having to
change for the afternoon because one therapist went home ill.
The group was beginning Chapter four. The facilitator had the chapter four booklets printed and
ready for the group. The facilitator reviewed the objectives and words to know with the youth. The
group also reviewed stinkin thinkin concepts and chapter three. The group completed exercises
one and two. The facilitator provided feedback for the role plays conducted during group. There
were no significant behavioral issues, however one youth was disruptive throughout group. The
youth was given several warnings. The facilitator and case management supervisor spoke with
this reviewer about this particular youth and have been attempting to work with this youth on
behaviors.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover of Impact of Crime facilitators during the twelve (12)-month
review period. The previous case management supervisor and three case managers left
employment during the annual review period. Two of the four facilitators left during a cohort and
those groups were not being co-facilitated. The other two staff left between cohorts. Should an
employee leave during a cohort and the group is not being co-facilitated, another trained facilitator
will continue with the group.
A review of sign-in sheets revealed there were no gaps in service. The program completed ten
(10) cohorts during the annual review period and were running two groups at the time of this
review. Each group had between eight and ten (10) youth.
The first cohort started March 16, 2021 and ended on July 1, 2021. The group started with eight
youth and five youth successfully completed the group. Three youth were released from the
program prior to successfully completing the group.
The second cohort started April 6, 2021 and ended on July 22, 2021. The group started with eight
youth and eight youth successfully completed the group. The group was held two days a week
from 4:00pm to 5:00pm. There were four sign-in sheets with no thave documented for the group.
The third cohort started May 17, 2021 and ended on September 6, 2021. The group started with
eight youth and six youth successfully completed the group. Two youth were released early from
the program and did not successfully complete. One youth was added to the group who was a
transfer from another program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The fourth cohort started June 29, 2021 and ended on October 7, 2021. The group started with
eight youth and eight youth successfully completed. The facilitator was out during the month of
August on medical leave. The case management superivsor facilitated the group during that
month.
The fifth cohort started July 5, 2021 and ended on September 3, 2021. The group started with two
youth and two youth successfully completed. The original facilitator left employment during the
cohort and the case management supervisor completed the cohort.
The sixth cohort started on October 14, 2021 and ended on January 19, 2022. The group started
with eight youth and four youth successfully completed. Four youth left the program and did not
successfully complete the IOC group. The group was held on Mondays and Wednesdays from
4:00pm to 5:00pm.
The seventh cohort started on October 5, 2021 and ended on January 20, 2022. The group
started with eight youth and two youth successfully completed. Six youth were released from the
program prior to successfully completing the group. This group was held on Tuesdays and
Thursdays. The group times varied, according to a review of the sign-in sheets.
The eighth cohort started October 19, 2021 and ended on January 13, 2022. The groups started
with eight youth and eight youth successfully completed. Eleven of the sessions were held for one
and a half hours.
The ninth cohort started on December 28, 2021 and ended on April 7, 2022. Ten (10) youth
started the group and seven youth successfully completed. Three youth were released from the
program prior to successfully completing the IOC group. This group was held on Tuesdays and
Thursdays from 3:00pm to 4:00pm.
The tenth cohort started December 28, 2021 and ended on March 24, 2022. The group started
with ten (10) youth and ten (10) youth. The group was held on Tuesdays and Thursdays from
3:30pm to 4:30pm. Groups on February 8, 2022, February 26, 2022, and March 18, 2022 were
held for two hours.
The eleventh cohort started January 31, 2022 and was ongoing at the time of the annual review.
The group started with nine youth and all nine youth were still in the group. The group is being
held on Mondays and Wednesdays from 3:30pm to 4:30pm. The group time changed in mid-
Feburary to 4:00pm to 5:00pm. One youth missed three sessions of the group due to being
transported for court. The facilitator completed individual sessions with this youth and those sign-
in sheets were provided for review.
The twelveth cohort started on February 22, 2022 and was ongoing at the time of the annual
review. The group started with twelve (12) youth and there were eleven (11) youth still enrolled at
the time of this review. One youth was transferred to another program due to behavioral issues.
Three sign-in sheets did not have signatures from the youth. This reviewer checked with the
facilitator about this. She could not remember why the youth did not sign the forms. The facilitator
did provide this reviewer the packets of worksheets completed for chapter six. Groups on April 12,
2022, April 19, 2022, and April 21, 2022 were held for two hours.
The thirteenth cohort started on April 19, 2022 and was ongoing at the time of the annual review.
The group started with seven youth and there continues to be seven youth at the time of the
review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process is for the case management supervisor to conduct internal fidelity
monitoring for the Impact of Crime groups. There were seven facilitators during the annual review
period. One of the facilitators did not have internal fidelity monitoring due to only facilitating two
groups before leaving employment.
Thirty-five (35) out of thirty-seven (37) internal fidelity monitoring observations were completed
during the annual review period. One facilitator was missing two months of fidelity monitoring
observations. This reviewer spoke with the case management supervisor about the missing
fidelity. The case management supervisor stated the missing fidelity was prior to her starting
employment as the supervisor. The previous case management supervisor did not complete the
observations for the two months.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The case management supervisor addresses any significant issues or corrective actions for the
Impact of Crime facilitators. The internal fidelity monitoring forms include a section to document
any significant issues or corrective actions.
A review of the internal fidelity forms and an interview with the case management supervisor
confirmed there were no significant issues or corrective actions during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided annual performance evaluations for the Impact of Crime (IOC) facilitators.
Three IOC facilitators received their annual performance evaluations and those were provided for
review. Two facilitators have not been at the program long enough to have received an annual
performance evaluation and two other facilitators are no longer employed at the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The annual performance evaluations do not include a section which includes an evaluation of
evidenced based service facilitation. This reviewer spoke with the case management supervisor
and suggested an addendum could be added to the performance evaluation to include evidenced
based group facilitation.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 16 weeks for this type of service. Of the 75 youth in the
sample, 29% (22 of 75) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 75 youth in the
sample, 85% (64 of 75) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 89%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 9 youth
Moderate-High = 21 youth
High = 37 youth
Total Youth in Sample = 75 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 37 youth
Total Youth in Sample = 75 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 89% (67 of 75) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 49% (37 of 75) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Okaloosa Youth
Academy.
75 youth completed the program during the review period. Of those youth, all 75 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 75 total youth sampled, only
66 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 22
received at least the recommended weeks of service. Youth in the sample completed between 8
and 16 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10
The program earned 8 points for Amount of Service: Contact Hours. Of the 75 total youth
sampled, all 75 included dosage in the EBS Module. Of those youth, 64 received at least the
recommended hours of service. Youth in the sample completed between 16 and 35 hours of
service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, fidelity monitoring is completed every month for each
facilitator, each facilitator’s performance evaluation includes discussion of their skill in delivering
this primary service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 11

source: DJJ

report_type: SPEP

file_name: FY2021-22 OYA IOC.pdf

program_name_raw: Okaloosa Youth Academy

program_status_note: Open

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Period: May 1, 2022 - April 30, 2023
SPEP Review Conducted: August 17, 2023
Florida Department of Juvenile Justice
Report Date: 10/18/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. There is a formal training
protocol for IOC, delivered by a qualified training. Case management staff are trained to facilitate
IOC groups. At the time of the review, five staff were trained to facilitate IOC: one case
management supervisor and four case managers. The four case management staff facilitated IOC
groups; the case management supervisor conducted internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the current version of the Impact of Crime (IOC) curriculum. Impact of Crime
groups are held on Mondays and Wednesdays or Tuesdays and Thursdays in the afternoons.
The program maintains the documentation for IOC groups in a binder kept in the case
management supervisor's office. The binder contains sign-in sheets, internal fidelity monitoring
documentation, and training documentation. All youth in the program are eligible to attend IOC
groups. Impact of Crime groups are closed groups, meaning no youth may be added to the group
once the initial lesson has been completed.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Impact of Crime was unable to be conducted during the review period;
therefore, this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
If the program experiences turnover, the program's policy is for another trained case manager to
conduct the group. The program did not experience turnover during the review period. All sign-in
sheets were provided for review. There were no gaps in service. All groups had between seven
and twelve (12) youth.
The program conducted ten (10) cohorts during the review period, with two additional ongoing at
the end of the cycle. The first cohort was started on January 31, 2022 with nine youth and ended
on June 1, 2022 with all nine youth. A second cohort started February 22, 2022 with twelve (12)
youth and ended on May 10, 2022 with eleven (11) youth. One youth was transferred from the
program due to behavioral issues.
A third cohort was started on April 12, 2022 with seven youth and ended on July 12, 2022 with all
seven youth. Groups were held for two hours on June 2nd, June 7th, and June 9th.
A fourth cohort started on April 19, 2022 with seven youth and ended on July 3, 2022 with all
seven youth. Groups from May 31, 2022 to July 3, 2022 were held for two hours two times a
week. The rest of the groups for the cohort were held for one hour, two times a week.
The fifth cohort began June 28, 2022 with eleven (11) youth and ended September 29, 2022 with
ten (10) youth. This group was co-facilitated. One youth was released prior to the full group
ending; however, the facilitator conducted individual sessions with the youth to ensure he
completed the curriculum. One youth was transferred from the program due to severe behavioral
issues. There were no gaps in service for this cohort.
The sixth cohort started on August 2, 2022 with nine youth and ended October 4, 2022 with all
nine youth. This timeframe does not meet the dosage for the IOC curriculum. One youth left the
program prior to group completion; however, the facilitator conducted individual sessions with this
youth. These groups were held for either one and a half hours or two hours each session. In order
to meet the required dosage, the group must be held for one hour two times a week. There were
no gaps in service during this cohort. The tracking log implemented during the first quarter was
reviewed. The tracking of the youth and groups appears to be working, due to more youth being
able to fully complete the IOC curriculum.
The seventh cohort started on October 6, 2022with twelve (12) and ended on December 8, 2022
with all twelve (12) youth. The eighth cohort began November 1, 2022 with eight and ended on
January 20, 2023 with all eight youth. These two cohorts were held on a shortened schedule to
ensure all youth were able to complete the curriculum prior to leaving the program. These were
the last of the youth the program was working to catch up in order to complete the curriculum.
The ninth and tenth cohorts were run simultaneously and began December 13, 2022 with eight
youth. Both cohorts ended March 30, 2023 with eight youth.
The two cohorts ongoing at the end of the review period started February 14, 2023 with nine
youth each. One still had nine youth, the other had eight remaining.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The case management supervisor conducts internal fidelity monitoring observations for the
facilitators of Impact of Crime (IOC). All fidelity forms were provided for review. They were
completed correctly, and all facilitators were provided feedback. There was one missing internal
fidelity monitoring form during the first quarter of the review period; all other necessary internal
fidelity monitorings were completed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The case management supervisor addresses any significant issues or corrective actions with the
facilitators for Impact of Crime. There is a section on the fidelity monitoring forms for
documentation of any issues. Upon review of these forms, there were no issues or corrective
actions noted for taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations for the review period do not include a section which includes
an evaluation of evidence-based service facilitation, to include Impact of Crime.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 82 youth in the
sample, 4% (3 of 82) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 82 youth in the
sample, 100% (82 of 82) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 93%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 11 youth
Moderate-High = 28 youth
High = 37 youth
Total Youth in Sample = 82 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 37 youth
Total Youth in Sample = 82 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 93% (76 of 82) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 45% (37 of 82) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 63 85 74%
This SPEP report evaluates Impact of Crime, an intervention delivered at Okaloosa Youth
Academy.
86 youth completed the program during the review period. Of those youth, 82 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 82 total youth sampled, all
82 included dosage with end dates in the EBS Module. Of those youth, 3 received at least the
recommended weeks of service. Youth in the sample completed between 10 and 16 weeks of
service, with an average of 12 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 82 total youth
sampled, all 82 included dosage in the EBS Module. Of those youth, 82 received at least the
recommended hours of service. Youth in the sample completed between 31 and 34 hours of
service, with an average of 32 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator and each facilitator’s performance
evaluation includes discussion of their skill in delivering this primary service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 Okaloosa YA IOC.pdf


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Okaloosa Youth Academy
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Seven Challenges
SPEP Review Period: May 1, 2022 - April 30, 2023
SPEP Review Conducted: August 17, 2023
Florida Department of Juvenile Justice
Report Date: 10/18/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes The Seven Challenges as a primary service for the substance abuse
component of the program. All therapy staff are trained to facilitate these groups. All training
records were provided for review. During the review period, there were six trained staff: one
clinical director and five therapists.
The clinical director or another trained therapist conducts internal fidelity monitoring for the
facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the current Seven Challenges manual and student journals. The manual
contains all lessons, worksheets, and a generalized script for the facilitators. The youth have a
separate workbook for this curriculum they work in throughout their time in the group.
Youth who have a substance abuse/use history and/or have a substance abuse diagnosis qualify
for this service. All sign-in sheets are maintained in a binder kept in the clinical director's office.
Seven Challenges groups are held Mondays, Wednesdays, and Fridays, from 2:15 to 3:15. The
groups are held as open groups, meaning the youth may enter the group upon admission to the
program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
An observation of Seven Challenges groups was conducted on Wednesday, February 22, 2023
from 2:00pm to 2:45pm. There were seven youth, one facilitator, and one staff member present in
the group. This session was a journaling day, however most of the class have completed all
Seven Challenges and are close to graduation. The facilitator decided to have a review day and
open discussion for the newer youth in the group. All youth participated in the group with
appropriate discussions. The facilitator continually brought the conversation back to one of the
Seven Challenges and asked thought-provoking questions for the group. There were no
behavioral issues during the group.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol, in the event of facilitator turnover, is for another trained staff to conduct
The Seven Challenges. The program experienced turnover in therapy staff. Three therapists and
one clinical director left during the review period; two were facilitating Seven Challenges groups at
the time of their departure. Groups were transitioned to another facilitator without a gap in service.
All sign-in sheets were provided for review. The Seven Challenges is held as an open group and
there are between eight and ten (10) youth in each group. Two groups are held each scheduled
day.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director conducts internal fidelity observations for The Seven Challenges facilitators. If
the clinical director conducts a group or is unavailable another trained staff will conduct the
observations. Due to significant turnover, no internal fidelity monitoring occurred during the last six
months of the review period. Following the end of the review period, the staffing challenges were
resolved, and fidelity monitoring has resumed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The clinical director addresses any significant issues or corrective actions with the facilitators for
The Seven Challenges groups. The internal fidelity forms include a section to document any
significant issues or corrective actions. A review of the completed internal fidelity forms confirmed
there were no corrective actions noted nor taken.
Internal fidelity monitoring was not conducted during the last six months of the review period;
therefore, points may not be awarded for this section. Internal fidelity monitoring must take place
regularly to receive points in this section.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations do not include a section which includes an evaluation of
evidence-based service facilitation, to include Seven Challenges.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 24 weeks for this type of service. Of the 36 youth in the
sample, 31% (11 of 36) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 36 youth in the
sample, 86% (31 of 36) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 19 youth
Total Youth in Sample = 36 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 19 youth
Total Youth in Sample = 36 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (34 of 36) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 53% (19 of 36) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 73 95 77%
This SPEP report evaluates Seven Challenges, an intervention delivered at Okaloosa Youth
Academy.
86 youth completed the program during the review period.
The program scored Medium for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 36 total youth sampled, only
31 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 11
received at least the recommended weeks of service. Youth in the sample completed between 19
and 31 weeks of service, with an average of 23 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 36 total youth
sampled, all 36 included dosage in the EBS Module. Of those youth, 31 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 65 and 100 hours of
service, with an average of 94 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Okaloosa Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups, fidelity monitoring is completed every month for each
facilitator, any corrective action needed is identified, applied, and demonstrated, and each
facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Okaloosa Youth Academy can optimize their SPEP Amount of Service score by ensuring dosage
for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted dosage
of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 Okaloosa YA 7C.pdf

service_type: Seven Challenges