summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Sex Offender Program
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Period: May 1, 2022 - April 30, 2023
SPEP Review Conducted: August 17, 2023
Florida Department of Juvenile Justice
Report Date: 10/18/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service. There is a formal training
protocol for IOC, delivered by a qualified training. Case management staff are trained to facilitate
IOC groups. At the time of the review, five staff were trained to facilitate IOC: one case
management supervisor and four case managers. The four case management staff facilitated IOC
groups; the case management supervisor conducted internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program is using the current version of the Impact of Crime (IOC) curriculum. Impact of Crime
groups are held on Mondays and Wednesdays or Tuesdays and Thursdays in the afternoons.
The program maintains the documentation for IOC groups in a binder kept in the case
management supervisor's office. The binder contains sign-in sheets, internal fidelity monitoring
documentation, and training documentation. All youth in the program are eligible to attend IOC
groups. Impact of Crime groups are closed groups, meaning no youth may be added to the group
once the initial lesson has been completed.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Impact of Crime was unable to be conducted during the review period;
therefore, this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
If the program experiences turnover, the program's policy is for another trained case manager to
conduct the group. The program did not experience turnover during the review period. All sign-in
sheets were provided for review. There were no gaps in service. All groups had between seven
and twelve (12) youth.
The program conducted ten (10) cohorts during the review period, with two additional ongoing at
the end of the cycle. The first cohort was started on January 31, 2022 with nine youth and ended
on June 1, 2022 with all nine youth. A second cohort started February 22, 2022 with twelve (12)
youth and ended on May 10, 2022 with eleven (11) youth. One youth was transferred from the
program due to behavioral issues.
A third cohort was started on April 12, 2022 with seven youth and ended on July 12, 2022 with all
seven youth. Groups were held for two hours on June 2nd, June 7th, and June 9th.
A fourth cohort started on April 19, 2022 with seven youth and ended on July 3, 2022 with all
seven youth. Groups from May 31, 2022 to July 3, 2022 were held for two hours two times a
week. The rest of the groups for the cohort were held for one hour, two times a week.
The fifth cohort began June 28, 2022 with eleven (11) youth and ended September 29, 2022 with
ten (10) youth. This group was co-facilitated. One youth was released prior to the full group
ending; however, the facilitator conducted individual sessions with the youth to ensure he
completed the curriculum. One youth was transferred from the program due to severe behavioral
issues. There were no gaps in service for this cohort.
The sixth cohort started on August 2, 2022 with nine youth and ended October 4, 2022 with all
nine youth. This timeframe does not meet the dosage for the IOC curriculum. One youth left the
program prior to group completion; however, the facilitator conducted individual sessions with this
youth. These groups were held for either one and a half hours or two hours each session. In order
to meet the required dosage, the group must be held for one hour two times a week. There were
no gaps in service during this cohort. The tracking log implemented during the first quarter was
reviewed. The tracking of the youth and groups appears to be working, due to more youth being
able to fully complete the IOC curriculum.
The seventh cohort started on October 6, 2022with twelve (12) and ended on December 8, 2022
with all twelve (12) youth. The eighth cohort began November 1, 2022 with eight and ended on
January 20, 2023 with all eight youth. These two cohorts were held on a shortened schedule to
ensure all youth were able to complete the curriculum prior to leaving the program. These were
the last of the youth the program was working to catch up in order to complete the curriculum.
The ninth and tenth cohorts were run simultaneously and began December 13, 2022 with eight
youth. Both cohorts ended March 30, 2023 with eight youth.
The two cohorts ongoing at the end of the review period started February 14, 2023 with nine
youth each. One still had nine youth, the other had eight remaining.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The case management supervisor conducts internal fidelity monitoring observations for the
facilitators of Impact of Crime (IOC). All fidelity forms were provided for review. They were
completed correctly, and all facilitators were provided feedback. There was one missing internal
fidelity monitoring form during the first quarter of the review period; all other necessary internal
fidelity monitorings were completed.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The case management supervisor addresses any significant issues or corrective actions with the
facilitators for Impact of Crime. There is a section on the fidelity monitoring forms for
documentation of any issues. Upon review of these forms, there were no issues or corrective
actions noted for taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations for the review period do not include a section which includes
an evaluation of evidence-based service facilitation, to include Impact of Crime.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 17 youth in the
sample, 76% (13 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 2 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 8%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 41%
Moderate to High Score: 2 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 17%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 0 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 41% (7 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 2 points.
Of the SPEP sample, 0% (0 of 17) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 2 25 8%
Totals 48 85 56%
This SPEP report evaluates Impact of Crime, an intervention delivered at Crestview Sex Offender
Program.
17 youth completed the program during the review period. Of those youth, all 17 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 17 total youth sampled, only
16 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 13
and 18 weeks of service, with an average of 17 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth, 17 received at least the
recommended hours of service. Youth in the sample completed between 32 and 33 hours of
service, with an average of 33 hours.
The program was awarded 2 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Based on the risk level of the youth placed at the program (8% Program Optimization
Percentage), the Department should work to ensure a larger proportion of higher risk youth are
recommended. As the program serves youth with a history of sexual behavior problems, this
includes a multitier strategy of working with stakeholders to achieve that practice, as well as
ensuring appropriate treatment services are available in the community to reduce the reliance on
residential commitment for lower risk youth presenting with sexually-related offenses.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 84%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Crestview Sex Offender Program can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator and each facilitator’s
performance evaluation includes discussion of their skill in delivering this primary service.
Crestview Sex Offender Program can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 CSOP IOC.pdf


state: FL

facility_name: Crestview Youth Academy

summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Sex Offender Program
Gulf Coast Youth Services
(Contract Provider)
4445 Straightline Road
Crestview, Florida 32539
Primary Service: Pathways/Roadmaps to Recovery
SPEP Review Period: May 1, 2022 - April 30, 2023
SPEP Review Conducted: August 17, 2023
Florida Department of Juvenile Justice
Report Date: 10/18/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Pathways/Roadmaps to Recovery.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Pathways/Roadmaps as a primary service for the youth in the program. The
therapy staff are trained to conduct Pathways/Roadmaps groups. All training records were
provided for review. During the review period there were three trained staff members: one
therapist, the clinical director, and the assistant clinical director. There was also one part-time
licensed therapist who trained to conduct groups and/or complete internal fidelity monitoring at a
point during the review period.
There is one main facilitator; the clinical director conducts internal fidelity.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided the Pathways/Roadmaps manual for review. The manual is a guided
workbook the youth work through on their own with a therapist. There are thirteen (13) chapters
with twelve (12) steps. There is not a script for the facilitator to follow. The manual is broken into
individual assignments and reading passages.
The groups are held as open groups, meaning the youth may enter the group upon admission to
the program. The sign-in sheets are maintained in the clinical director's office. All youth in the
program are eligible to attend the Pathways groups. Groups are held Tuesdays, Thursdays, and
Fridays at 2:15pm.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
A session of Pathways/Roadmaps was observed on Tuesday, November 15, 2022 from 2:15pm
to 3:15pm. There were seven youth, one facilitator, and one direct care staff member in group.
The facilitator shared the lesson with this reviewer. The facilitator adhered to the lesson, was
knowledgeable of the material, and was comfortable with facilitating the lesson. All youth actively
participated in the lesson and completed the activity.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Pathways/Roadmaps is for another
trained staff to continue with the group. The program experienced turnover for the
Pathways/Roadmaps groups. One facilitator was brought in PRN to provide coverage during the
turnover. All sign-in sheets were provided for review and there were no gaps in service during the
review period. Pathways/Roadmaps is an open group. There were between six and ten (10) youth
in each of the two groups held consistently throughout the year.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director is to conduct internal fidelity monitoring for the Pathways/Roadmaps groups.
Three of six internal fidelity monitorings were conducted during the first quarter of the review.
Three months were missing for the clinical director. The clinical director the missing fidelity forms
were due to her having to co-facilitate with the newly trained staff.
There was one month of missing internal fidelity monitoring observations during the second
quarter of the review period. The interim clinical director facilitated groups during the month of
October while the primary facilitator was out of town. There was no one else trained at the
program to conduct internal fidelity monitoring observations.
No internal fidelity monitoring was conducted during the third or fourth quarters of the review
period.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any significant issues or corrective actions for the facilitators of
Pathways/Roadmaps. The completed internal fidelity monitoring forms include a section to
document any issues or corrective actions and included feedback for the facilitator. There were no
significant issues or corrective actions noted.
Internal fidelity monitoring was not conducted during the last six months of the review period;
therefore, points may not be awarded for this section. Internal fidelity monitoring must take place
regularly to receive points in this section.
The program can earn 1 point if any corrective action needed is identified, applied, and
demonstrated.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual performance evaluations for the review period do not include a section which includes
an evaluation of evidence-based service facilitation, to include Pathways/Roadmaps to Recovery.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 17 youth in the
sample, 76% (13 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 17 youth in the
sample, 100% (17 of 17) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 2 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 8%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 41%
Moderate to High Score: 2 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 17%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 0 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 0%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 0 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 41% (7 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 2 points.
Of the SPEP sample, 0% (0 of 17) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 2 25 8%
Totals 58 95 61%
This SPEP report evaluates Pathways/Roadmaps to Recovery, an intervention delivered at
Crestview Sex Offender Program.
17 youth completed the program during the review period. Of those youth, all 17 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 17 total youth sampled, only
13 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 13
received at least the recommended weeks of service. Youth in the sample completed between 35
and 53 weeks of service, with an average of 42 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, all 17 included dosage in the EBS Module. Of those youth, 17 received at least the
recommended hours of service. Youth in the sample completed between 52 and 154 hours of
service, with an average of 90 hours.
The program was awarded 2 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Based on the risk level of the youth placed at the program (8% Program Optimization
Percentage), the Department should work to ensure a larger proportion of higher risk youth are
recommended. As the program serves youth with a history of sexual behavior problems, this
includes a multitier strategy of working with stakeholders to achieve that practice, as well as
ensuring appropriate treatment services are available in the community to reduce the reliance on
residential commitment for lower risk youth presenting with sexually-related offenses.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 85%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Crestview Sex Offender Program can optimize their SPEP Quality of Service Delivery score by
ensuring facilitator turnover does not impact groups, fidelity monitoring is completed every month
for each facilitator, any corrective action needed is identified, applied, and demonstrated, and
each facilitator’s performance evaluation includes discussion of their skill in delivering this primary
service.
Crestview Sex Offender Program can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2022-23 CSOP Pathways SO 6-12.pdf

program_name_raw: Crestview Sex Offender Program

program_status_note: Closed

service_type: Pathways to Self‐Discovery and Change


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Youth Academy Nonsecure
Youth Opportunity Investments, LLC
(Contract Provider)
4449 Straightline Road
Crestview, Florida 32539
Primary Service: Impact of Crime
SPEP Review Period: February 15, 2022 - February 14, 2023
SPEP Review Conducted: February 23, 2023
Florida Department of Juvenile Justice
Report Date: 07/31/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime (IOC) as a primary service for all youth in the program.
Impact of Crime: Addressing the Harm is an integrated, cognitive behavioral approach designed
to use with adjudicated youth in residential treatment programs. This curriculum uses cognitive
restructuring, social skills development, and development of peaceful conflict resolution skills.
The case management staff are formally trained to facilitate IOC groups. During the annual review
year, six staff were trained to facilitate IOC groups: five case managers and one director of case
management. Three of the six staff facilitated IOC groups for the program. The director of case
management conducts internal fidelity monitoring observations.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the current Impact of Crime (IOC) manual at the program. The manual
includes implementation guidelines, identifies the order the lessons should be delivered, includes
detailed instructions for lesson delivery, and all needed worksheets for the youth. The curriculum
includes separate youth workbooks.
Impact of Crime groups are held on Tuesdays and Thursdays from 5:00pm to 6:00pm. The
documentation for IOC is kept in a binder maintained in the case management director's office.
The binder includes sign-in sheets and internal fidelity monitoring documentation. Impact of Crime
are closed groups meaning youth cannot be added to the group once the first lesson has been
completed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime group was observed on Wednesday, November 16, 2022. In attendance were
eleven (11) youth, one staff, and the facilitator. The lesson for the day was a review of chapters
one through seven. The reviewer was provided a copy of the review packet. The last session of
this cohort is scheduled for November 23, 2022. The group will conduct their restorative justice
project during the week of November 21, 2022. The group project is washing and cleaning cars of
the employees and the program vans.
The facilitator created a review packet for the youth to complete. The group went through it
together to prepare for their final lesson. All the youth participated and they all got their answers
correct. The youth were engaged during the whole process.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of facilitator turnover for Impact of Crime (IOC) is for another
trained staff to continue with group facilitation. The program experienced facilitator turnover for the
annual review period. Two case managers and two directors of case management left
employment. All staff left between cohorts, therefore the groups were not effected.
All sign-in sheets were provided for review. The program completed four cohorts of IOC during the
annual review period. The first cohort started January 4, 2022 and ended on April 26, 2022. The
group began with ten (10) youth and ten (10) youth successfully completed the group. This group
was held on Tuesdays and Thursdays from 2:30pm to 3:30pm. This group was held consistent to
the schedule.
The second cohort was started on April 8, 2022 and ended on July 28, 2022. The cohort started
with ten (10) youth and ten (10) youth successfully completed the cohort. This group was held on
Tuesdays and Thursdays on a consistent basis. The facilitator also kept detailed notes on each
sign-in sheets.
The third cohort started on June 28, 2022 and ended on October 5, 2022. The program began
with five youth and five youth successfully completed the group. The group was held on Tuesdays
and Thursdays from 2:00pm and 3:00pm. All of the sign-in sheets included a description of the
lesson and provided great detail on all of them.
The fourth cohort started on September 20, 2022 and ended on January 10, 2023. Six youth
began the group and six youth successfully completed the group. This group was held
consistently to the schedule.
A fifth cohort began on December 13, 2021 and was continuing at the time of the review. There
were seven youth in the group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The director of case management conducts internal fidelity monitoring for Impact of Crime (IOC)
facilitators. There were six staff who facilitated IOC cohorts during the annual review period.
The following internal fidelity monitoring observations were missing: two in March 2022, two in
April 2022, one in June 2022, one in September 2022, and one in January 2022. All other internal
fidelity forms were provided for review.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The case management director addresses any significant issues or corrective actions for the
facilitators of Impact of Crime (IOC).
All internal fidelity monitoring forms were provided for review. All facilitators did receive feedback
and this was documented on the fidelity forms. There were no documented significant issues or
corrective actions for the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluation includes what evidence-based services the staff
are trained in and are evaluated on facilitation.
The case management director is the only current staff still employed and has been at the
program for a year or longer. The case management director was evaluated for Impact of Crime
evaluation for the annual review year.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 14 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (29 of 29) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 48% (14 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 85 85 100%
This SPEP report evaluates Impact of Crime, an intervention delivered at Crestview Youth
Academy Nonsecure.
29 youth completed the program during the review period. Of those youth, all 29 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 29 total youth sampled, all
29 included dosage with end dates in the EBS Module. Of those youth, 29 received at least the
recommended weeks of service. Youth in the sample completed between 16 and 32 weeks of
service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, all 29 included dosage in the EBS Module. Of those youth, 29 received at least the
recommended hours of service. Youth in the sample completed between 32 and 33 hours of
service, with an average of 32 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Crestview Youth Academy Nonsecure can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator.
Crestview Youth Academy Nonsecure can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 CYA-NS IOC.pdf

program_name_raw: Crestview Youth Academy Nonsecure

program_status_note: Open

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Youth Academy Nonsecure
Youth Opportunity Investments, LLC
(Contract Provider)
4449 Straightline Road
Crestview, Florida 32539
Primary Service: Male Healthy Relationships
SPEP Review Period: February 15, 2022 - February 14, 2023
SPEP Review Conducted: February 23, 2023
Florida Department of Juvenile Justice
Report Date: 07/31/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Male Healthy Relationships.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Male Healthy Relationships (MHR) as a primary service for all the youth in
the program. Male Healthy Relationships is a combined curriculum of Teen Relationships and
Young Men's Work. Therapy staff conduct MHR groups.
During the annual review period, fourteen (14) staff were trained to facilitate MHR groups: two
clinical directors, one assistant clinical director, and twelve (12) therapists. Seven of the fourteen
(14) staff facilitated groups: five therapists, the clinical director, and the assistant clinical director.
The clinical director and/or the assistant clinical director conducts internal fidelity monitoring for
MHR groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
For Male Healthy Relationships, the program maintains two manuals the therapists work from;
Young Men's Work and Teen Relationships. The Young Men's Work manual is maintained in a
binder in the clinical director's office. It is approximately ten (10) sessions and includes videos and
handouts. The manual is broken into individual sessions which include the objective, agenda,
session outline, and all handouts. The lessons also include a script to follow. The manual for Teen
Relationships is maintained in a binder in the clinical director's office. It is five main topics broken
down into individual lessons. All lessons and handouts are included. Facilitator guidelines are also
included, but no script is provided.
Teen Relationships is held on Tuesdays from 3:00pm to 4:00pm and Young Men's Work is held
on Fridays from 3:00pm to 4:00pm. The program maintains documentation in three different
binders kept in the clinical director's office. The first binder includes sign-in sheets for all groups
separated between Teen Relationships and Young Men's Work. The second binder includes all
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
internal fidelity monitoring for each facilitator separated by month. The third binder includes all
training documentation for the facilitators.
All youth are eligible to attend MHR groups in the program. The groups are open, meaning the
youth can join the groups upon admission to the program.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Teen Relationships was observed on-site on Wednesday, November 30, 2022 from
3:30pm to 4:30pm. There were eight youth, one facilitator, and one staff member in attendance.
The lesson for the group was The Relationship Bill of Rights. The youth were all engaged in the
discussion throughout the entire session. The facilitator was comfortable facilitating the lesson
and guiding the discussion. The facilitator asked thought-provoking questions for the youth and
was engaging for the full hour. The facilitator provided appropriate feedback to the youth.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process in the event of facilitator turnover for Male Healthy Relationships (MHR) is
for another trained staff to facilitate the group. The therapy staff at the program rotate facilitation
so the youth are familiar with the facilitators.
The program experienced turnover for MHR groups during the annual review period. Nine
therapists left employment during the review period. The program has been able to quickly hire
new therapy staff and the turnover did not affect the MHR groups.
All sign-in sheets were provided for review. There were no gaps in service during the annual
review period. All groups had no more than eight youth in each group. The program holds two to
three groups each scheduled day based upon the number of youth in the program.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process for monthly internal fidelity monitoring is for the clinical director and/or
assistant clinical director to conduct internal fidelity observations for the facilitators of Male
Healthy Relationships (MHR). Each facilitator is monitored two times a month, once for Teen
Relationships and once for Young Men's Work. Twelve (12) of the fourteen (14) trained staff
facilitated groups during the annual review period.
All internal fidelity monitoring forms were provided for review. There were some missing
observations during the annual review period: two for February 2022, three for March 2022, and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
three for April 2022. When the new clinical director was hired, all internal fidelity monitoring
observations were completed on a consistent basis.
The clinical director and assistant clinical director are both trained in the curricula.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for corrective actions is for the clinical director and/or the assistant clinical
director address any significant issues or corrective actions for the facilitators of Male Healthy
Relationships (MHR).
The internal fidelity forms include a section to document any corrective actions. All forms included
feedback for each facilitator. A review of the fidelity forms confirmed no corrective actions were
given during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A blank annual performance evaluation was provided for review. The evaluations include a
section to document what curriculum the facilitator provides and is evaluted for their proficiency.
No facilitators for Male Healthy Relationships have been at the program for the full year to receive
an annual performance evaluation, therefore this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 14 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (29 of 29) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 48% (14 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 95 95 100%
This SPEP report evaluates Male Healthy Relationships, an intervention delivered at Crestview
Youth Academy Nonsecure.
29 youth completed the program during the review period. Of those youth, all 29 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 29 total youth sampled, all
29 included dosage with end dates in the EBS Module. Of those youth, 29 received at least the
recommended weeks of service. Youth in the sample completed between 24 and 44 weeks of
service, with an average of 35 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, all 29 included dosage in the EBS Module. Of those youth, 29 received at least the
recommended hours of service. Youth in the sample completed between 46 and 78 hours of
service, with an average of 65 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Crestview Youth Academy Nonsecure can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator.
Crestview Youth Academy Nonsecure can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 CYA-NS MHR.pdf

service_type: Male Healthy Relationships


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Youth Academy Nonsecure
Youth Opportunity Investments, LLC
(Contract Provider)
4449 Straightline Road
Crestview, Florida 32539
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: February 15, 2022 - February 14, 2023
SPEP Review Conducted: February 23, 2023
Florida Department of Juvenile Justice
Report Date: 08/01/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Skillstreaming the Adolescent (Skillstreaming) as a primary service for all
youth in the program. All therapy staff are trained by the clinical director to conduct Skillstreaming
groups.
During the annual review period there were fifteen (15) staff trained to facilitate Skillstreaming
groups. All staff trained were therapy staff and all training records were provided for review.
Twelve (12) staff had the opportunity to facilitate Skillstreaming groups during the annual review
period. The clinical director or assistant clinical director conducts internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for this reviewer to
reference during the review. The program currently uses The Prepare Curriculum manual by
Arnold Goldstein, in conjunction with the Skillstreaming the Adolescent (Skillstreaming) skill cards.
The manual does not have a specific script for each lesson. It is divided into exercises for each
lesson. Each lesson also contains the objective, materials, procedure: guided questions and
discussion. Once the lesson is completed, the following exercise is practicing the skill using the
identified skill card.
Skillstreaming groups are held Tuesdays and Thursdays from 3:00pm to 4:00pm. The program
maintains documentation for the groups in three separate binders. One contains all sign-in sheets,
a second binder contains all fidelity monitoring forms, and a third binder contains all training
records. Skillstreaming is held as an open group, meaning the youth may enter the group upon
admission to the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Skillstreaming the Adolescent was held on November 29, 2022 from 3:30pm to
4:30pm. There were eight youth, one facilitator, and one staff in attendance. The lesson for the
group was Dealing with Accusations. The reviewer was provided a copy of the lesson. The
facilitator was familiar with the curriculum and could utilize their own words throughout the group
time. The facilitator ensured the youth understood and demonstrated the skill before moving on to
the next part of the lesson. All youth and the facilitator provided feedback for the role plays. All
youth came up with appropriate examples for their role plays.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of facilitator turnover for Skillstreaming the Adolescent
(Skillstreaming) is for another trained staff to continue with group facilitation. All therapists rotate
facilitation throughout the year and the youth are familiar with each.
The program experienced turnover with eight staff leaving employment during the annual review
period: seven therapists and one clinical director. Since the therapists rotate facilitation, this had
minimal impact to the youth. All sign-in sheets were provided for review and there were no gaps in
service during the annual review period. There were no more than eight youth in each group.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's process for internal fidelity monitoring observations for Skillstreaming the
Adolescent (Skillstreaming) is for the clinical director or assistant clinical director complete the
observations.
There were twelve (12) staff who facilitated groups during the annual review period. The internal
fidelity monitoring forms were provided for review. Five fidelity observations were missing for the
annual review period, two in March 2022 and three in April 2022. There was marked improvement
in fidelity observations when the new clinical director started employment.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's process for corrective actions is for the clinical director or assistant clinical director
addresses any significant issues or corrective actions for Skillstreaming the Adolescent
facilitators.
The internal fidelity forms include a section to document any corrective actions. A review of the
forms and a conversation with the clinical director confirmed there were no corrective actions
during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A blank annual performance evaluation was reviewed and included specific information about
group facilitation and delivery of the employee's specific intervention(s).
There were no current facilitators for Skillstreaming the Adolescent groups who had been
employed for a year or longer, therefore this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 29 youth in the
sample, 100% (29 of 29) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 5 youth
Moderate-High = 10 youth
High = 14 youth
Total Youth in Sample = 29 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 48%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 14 youth
Total Youth in Sample = 29 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (29 of 29) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 48% (14 of 29) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 85 85 100%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Crestview
Youth Academy Nonsecure.
29 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 29 total youth sampled, all
29 included dosage with end dates in the EBS Module. Of those youth, 29 received at least the
recommended weeks of service. Youth in the sample completed between 24 and 31 weeks of
service, with an average of 27 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 29 total youth
sampled, all 29 included dosage in the EBS Module. Of those youth, 29 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 46 and 50 hours of
service, with an average of 50 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Crestview Youth Academy Nonsecure can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator.
Crestview Youth Academy Nonsecure can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring tht youth receive the
full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 CYA-NS Skillstreaming.pdf

service_type: Skillstreaming the Adolescent


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Crestview Youth Academy Secure
Youth Opportunity Investments, LLC
(Contract Provider)
4449 Straightline Road
Crestview, Florida 32539
Primary Service: Seven Challenges
SPEP Review Period: February 15, 2022 - February 14, 2023
SPEP Review Conducted: February 23, 2023
Florida Department of Juvenile Justice
Report Date: 08/01/2023
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seven Challenges as a primary service. All therapy staff are trained to
facilitate the Seven Challenges groups. Ten (10) therapy staff were trained to facilitate the groups
during the annual review period. Of the ten (10), seven staff facilitated groups: five therapists, the
assistant clinical director, and the clinical director.
The clinical director and/or the assistant clinical director conducts internal fidelity monitoring for
Seven Challenges facilitators.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program currently uses the Seven Challenges manual by Rober Schwebel, PhD. The youth
also have workbooks to work through at their own pace. These workbooks include the lesson
readings and journaling pages. The facilitator has a Master manual which has all of the lessons.
Seven Challenges groups are held on Mondays at 2:00pm, Saturdays and Sundays in the
morning. This curriculum is eligible for youth who have a substance abuse diagnosis. Group times
vary based upon the weekend schedule of the youth. The documentation for the curriculum is
maintained in three binders kept in the clinical director's office. The first binder includes all sign-in
sheets, the second binder includes internal fidelity monitoring forms, and the third binder includes
all training documentation. Seven Challenges is held as an open group, meaning the youth may
enter the group upon admission to the program.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Seven Challenges was completed on Monday, November 28, 2022. There were
fifteen (15) youth, one facilitator, and two staff present for the group. The facilitator was
comfortable facilitating the group. This was a journaling day. The youth were all working in their
individual journals quietly. They were able to ask questions to the group or the facilitator. The
facilitator did a good job keeping the youth on track and assisting when needed.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's process in the event of facilitator turnover for Seven Challenges is for another
trained staff to facilitate the group. All therapy staff rotate group facilitation so the youth are
familiar with each facilitator.
The program experienced facilitator turnover for the Seven Challenges groups during the annual
review period. Nine therapists left employment. The program has been able to quickly hire new
therapy staff and the turnover did not affect the Seven Challenges groups.
All sign-in sheets were provided for review and there were no gaps in service during the annual
review period.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director and/or assistant clinical director conducts internal fidelity monitoring for the
facilitators of Seven Challenges. There were ten (10) therapy staff who facilitated Seven
Challenges during the annual review period.
All internal fidelity monitoring forms were provided for review. There were some missing
observations: two in February 2022, five in March 2022, two in April 2022, and one in June 2022.
Since the clinical director was hired, the internal fidelity monitoring have been consistently done.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director and/or assistant clinical director addresses any significant issues or corrective
actions with the facilitators for Seven Challenges. The internal fidelity forms include a section to
document any corrective actions needed for facilitators.A review of the forms confirmed there
were no corrective actions for any of the facilitators during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: N/A Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided an annual performance evaluation for review. The evaluations include a
section to document the evidence-based services the facilitator is trained in and evalutes their
proficiency in delivering Seven Challenges.
There were no staff in the program who had been there for a year or longer and received an
annual performance evaluation, therefore this indicator is rated as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 24 weeks for this type of service. Of the 33 youth in the
sample, 100% (33 of 33) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 33 youth in the
sample, 100% (33 of 33) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 5 youth
Moderate-High = 5 youth
High = 20 youth
Total Youth in Sample = 33 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 61%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 20 youth
Total Youth in Sample = 33 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (30 of 33) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 61% (20 of 33) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 23 25 92%
Totals 93 95 98%
This SPEP report evaluates Seven Challenges, an intervention delivered at Crestview Youth
Academy Secure.
34 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 33 total youth sampled, all
33 included dosage with end dates in the EBS Module. Of those youth, 33 received at least the
recommended weeks of service. Youth in the sample completed between 27 and 41 weeks of
service, with an average of 36 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 33 total youth
sampled, all 33 included dosage in the EBS Module. Of those youth, 33 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 80 and 100 hours of
service, with an average of 98 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Crestview Youth Academy Secure can optimize their SPEP Quality of Service Delivery score by
ensuring fidelity monitoring is completed every month for each facilitator.
Crestview Youth Academy Secure can optimize their SPEP Amount of Service score by ensuring
dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full targeted
dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 CYA-S 7C.pdf

program_name_raw: Crestview Youth Academy Secure

service_type: Seven Challenges