summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Period: May 15, 2020 to May 14, 2021
SPEP Review Conducted: May 18, 2021 to May 21, 2021
Florida Department of Juvenile Justice
Report Date: August 11, 2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime as a primary service. The program's documentation
reflected the program's protocol is for a licensed therapist or certified trainer to conduct Impact of
Crime groups.
The program has had seven trained staff during the twelve month review period; one clinical
coordinator, one transition specialist, and five case management staff. All training records were
provided for review and all staff were formally trained in the Impact of Crime curricula. Two staff
were trained June 2017, two staff were trained July 2019, one staff was trained in November
2019, and one staff was trained in November 2020. One employee who was previously trained left
employment during the twelve month review period.
There were four main facilitators for the Impact of Crime groups at the program during the twelve
month review period. All training records were provided and all staff were formally trained in the
Impact of Crime curriculum.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program’s facilitator utilizes the Impact of Crime (IOC), Addressing the Harm curriculum,
version 2.0. The program has a manual for the delivery of services and a workbook for the
students to use. The curriculum provides a script and instructions for seven chapters which
requires a minimum of thirty-two contact hours to facilitate. The curriculum includes
implementation guidelines, identifies the order the lessons are to be delivered, and provides
detailed instructions for the delivery of each lesson.
Impact of Crime groups are currently scheduled for Wednesday and Fridays from 4:00pm to
5:00pm for one hour. Eligibility for the Impact of Crime group is based upon the youth's
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
criminogenic needs identified on the Residential Assessment for Youth, in accordance with the
program's contract. The program maintains a binder with all documentation for Impact of Crime
groups kept in the clinical director's office.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was unable to be observed during the review period; ; therefore, this
indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol, in the event of turnover, is for another trained staff member to continue
with the current cohort. Two of the four cohorts of Impact of Crime held during this review period
were co-faciltiated. The facilitator who left during the review period did not impact any of the
groups being conducted. There were no gaps in service during the twelve month review period.
The program completed three cohorts during the twelve month review period.
The first cohort was started on February 4, 2020 and ended on May 26, 2020. The group began
with eight youth and ended with seven youth. There were no notes indicating as to why one youth
did not complete the group.
The second cohort was started on October 19, 2020 and ended on February 24, 2021. Three
youth began this group and three youth successfully completed this group. The cohort had a low
number of youth attending due to a low amount of youth in the program. The program received
permission from the Department of Juvenile Justice to conduct the cohort with a lower amount of
youth than is required. While there was a gap in services between May 26 and October 19, all
eligible youth attended Impact of Crime groups during the review period.
A third cohort started on January 19, 2021 and ended on May 18, 2021. This group started with
eight youth and ended with seven youth. One youth was transferred from the program, per
program documentation provided for review. The program also provided documentation of
community service conducted during this cohort. The youth were unable to leave the program due
to COVID protocols, therefore community service was completed in and around the program.
All sign-in sheets were provided for review. The sign-in sheets covered a description of the
lessons covered and information on the youth and faciltiators. Groups were held consistently for
one hour two times a week.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy for internal fidelity monitoring is, "Fidelity monitoring is conducted by trained
program personnel who have complete formal training in the evidence-based model being
monitored. Corporate management staff assists in all fidelity monitoring reviews. In no case will
program staff monitor their own services provision. However, in accordance with Department
standards, it is allowable for staff in the same program to monitor each other."
The internal fidelity monitoring forms for Impact of Crime are thorough and include feedback from
the observer, a section for corrective actions, and a section for behavioral issues addressed
during the group with the youth. All internal fidelity monitoring forms included the faciltator's name,
the observer's name, and the date and time of the group being observed. Other trained facilitators
conducted internal fidelity monitoring for the Impact of Crime groups held during the twelve month
review period.
There were four facilitators during this review period. Fifteen out of fifteen internal fidelity
monitorings were completed . All documentation was provided for review.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy for addressing corrective actions is "Corrective actions will be issued when
program are not conducting treatment as prescribed or when other problems are identified during
the monitoring process. The Chief Operations Officer will be responsible for ensuring that
problems identified are corrected in a timely manner."
A review of the internal fidelity monitoring forms indicated no corrective actions were given during
the review period for Impact of Crime facilitators.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitators' employee performance evaluations found an entry for evlaution of the
staff's performance in delivering Impact of Crime. Six out of the seven trained facilitators had
annual evaluations provided for review. One facilitator left employment prior to receiving the
annual performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 23 youth in the
sample, 74% (17 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 23 youth in the
sample, 96% (22 of 23) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 9 youth
High = 7 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 30%
High Score: 10 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 77%
Table 2
High = 7 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (20 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 30% (7 of 23) youth scored High-Risk to reoffend, for a score of 10 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 74 85 87%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment.
24 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 23 total youth sampled, all
23 included dosage with end dates in the EBS Module. Of those youth, 17 received at least the
recommended weeks of service. Youth in the sample completed between 4 and 37 weeks of
service, with an average of 17 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, all 23 included dosage in the EBS Module. Of those youth, 22 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 10 and 87 hours of
service, with an average of 40 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2020-21 JUST IOC.pdf


state: FL

facility_name: Juvenile Unit for Specialized Treatment (JUST)

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Thinking for a Change
SPEP Review Period: May 15, 2020 to May 14, 2021
SPEP Review Conducted: May 18, 2021 - May 21, 2021
Florida Department of Juvenile Justice
Report Date: August 11, 2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Thinking for a Change as a primary service. The training protocol for this
curriculum is for licensed therapy staff to conduct Thinking for a Change groups, in accordance
with the program's contract.
There are currently three trained staff members; one clinical director trained in February 2014 and
two clinical coordinators trained August 2017. One of the clinical coordinators is the primary
facilitator of Thinking for a Change groups. All training records were provided for review. All three
staff members were formally trained to facilitate the Thinking for a Change curriculum.
During the twelve month review period, two of the three trained staff members have conducted
Thinking for a Change groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the Thinking for a Change curriculum (T4C), version 4.0. The manual provides
a script for each of the twenty-five lessons. The youth are introduced to the skill and skill steps for
each lesson. There are facilitator notes located in the margins of each page. All skill cards, slides,
and homework sheets are available electronically, and each facilitator has access to the materials
required to conduct T4C. The manual and supplemental materials are detailed. Documentation for
the primary service is housed in a binder which is organized and easy to locate. Cohorts are
stapled separately for easy review.
Thinking for a Change is currently held on Tuesdays and Thursdays from 4:00pm to 5:00pm.
Eligibility for the Thinking for a Change group is based upon the youth's criminogenic needs
identified on the Residential Assessment for Youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed on May 27, 2021 at 2:10pm and lasted
approximnately one hour. Present for group were nine youth, one direct care staff, the clinical
director, and the facilitator. Group began with a review of the previous lesson and assigned
homework. The facilitator enlisted two youth to replay the previous session's role-play to reinforce
concepts of the difficult lesson. The group then transitioned to lesson 21, Thinking for Choices
and Consequences. To wrap up, the facilitator and clinical director asked the youth to take turns
recapping the previous chapters (17-21). The facilitator used real life scenarios to help make the
material more relatable for the youth. He followed the lesson plan and was kowledgeable of the
material.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy, in the event of turnover, is for another trained facilitator to continue with the
current cohort. The groups are not always co-facilitated. During this review period, one cohort was
co-facilitated and two were not. The program did not experience any turnover during the twelve
month review period for the current Thinking for a Change (T4C) facilitators.
There was one gap from May 28, 2020 to September 23, 2020. This gap was not caused by staff
turnover. The program experienced a significant lack of youth entering the program during this
timeframe. This does not qualify as an unnecessary gap in service. All youth in the program who
were eligible for this service were able to complete this service before their discharge.
The program completed two cohorts of T4C. All sign-in sheets were provided for review.
One group started on January 27, 2020 and ended on May 28, 2020. This cohort is included
because it ended at the start of the review period for the 2020-2021 review year. Eight youth
started this group and eight youth successfully completed.
The second group started September 23, 2020 and ended on February 5, 2021. The group
started with six youth and ended with five youth. The sign-in sheets provided by the program for
review had documentation one youth was released prior to the end of the group. However, this
youth received one-to-one sessions and was able to successfully complete the Thinking for a
Change curriculum.
A third cohort started on March 16, 2021 and is continuing to meet. The group started with nine
youth and continues with the same nine youth at the time of the annual review.
The program provided documentation describing how the youth participated in each group and
included a description of the lesson completed. The program also provided sign-in sheets for
make-up sessions if youth were gone from the program during the scheduled group time.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program's policy for internal fidelity monitoring is, "Fidelity monitoring is conducted by trained
program personnel who have complete formal training in the evidence based model being
monitored. Corporate management staff assists in all fidelity monitoring reviews. In no case will
program staff monitor their own services provision. However, in accordance with Department
standards, it is allowable for staff in the same program to monitor each other."
The clinical director is designated to conduct internal fidelity monitoring for Thinking for a Change
groups. Two facilitators conducted Thinking for a Change groups during the twelve month review
period. All internal fidelity monitoring forms were provided for this review. Thirteen out of eleven
required internal fidelty monitorings were conducted during this review period. One faciltiator
received two additional monitorings. The internal fidelity monitoring forms are thorough and
include feedback from the observer, a section for corrective actions, and a section for behavioral
issues addressed during the group with the youth. All internal fidelity monitoring forms included
the faciltator's name, the observer's name, and the date and time of the group being observed.
One internal fidelity monitoring form dated April 13, 2021 did not include a facilitator name,
however, there was only one facilitator during that timeframe. No groups were held between May
28, 2020 and September 23, 2020.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy for addressing corrective actions is "Corrective actions will be issued when
programs are not conducting treatment as prescribed or when other problems are identified during
the monitoring process. The Chief Operations Officer will be responsible for ensuring that
problems identified are corrected in a timely manner."
A review of the internal fidelity monitoring forms for the twelve month review period determined
there were no corrective actions noted nor taken for Thinking for a Change.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitators' employee performance evaluations found an entry for evlaution of the
staff's performance in delivering Thinking for a Change. Two of two facilitators received their
annual performance evaluation at the time of this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 11 youth in the
sample, 91% (10 of 11) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 45 contact hours for this type of service. Of the 11 youth in the
sample, 100% (11 of 11) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 18 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 72%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 91%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 2 youth
Moderate-High = 5 youth
High = 3 youth
Total Youth in Sample = 11 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 27%
High Score: 8 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 62%
Table 2
High = 3 youth
Total Youth in Sample = 11 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 91% (10 of 11) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 27% (3 of 11) youth scored High-Risk to reoffend, for a score of 8 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 18 25 72%
Totals 91 100 91%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Juvenile Unit for
Specialized Treatment.
24 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 11 total youth sampled, all
11 included dosage with end dates in the EBS Module. Of those youth, 10 received at least the
recommended weeks of service. Youth in the sample completed between 5 and 22 weeks of
service, with an average of 18 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 11 total youth
sampled, all 11 included dosage in the EBS Module. Of those youth, 11 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 47 and 57 hours of
service, with an average of 51 hours.
The program was awarded 18 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 98%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

file_name: FY2020-21 JUST T4C.pdf

program_name_raw: Juvenile Unit For Specialized Treatment

program_status_note: Open

service_type: Thinking for a Change


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Period: November 1, 2021 - October 31, 2022
SPEP Review Conducted: December 13, 2022
Florida Department of Juvenile Justice
Report Date: 2/20/23
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Imapct of Crime (IOC) as a primary service. Case management staff and
therapy staff are trained to facilitate IOC groups. During the twelve (12)-month review period,
twelve (12) staff were trained to facilitate IOC groups. All training records were provided for review
and all staff were trained formally by qualified traininers.
During the review period, six of the twelve (12) trained staff conducted IOC groups. The trained
facilitators monitor each other on a monthly basis for internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the current version of the Impact of Crime (IOC) manual. The manual
includes detailed lesson plans, a script for the facilitator, facilitator notes, and implementation
information. The manual also includes workbooks for the youth. All facilitators have the newest
version of the manuals.
The program holds IOC groups two days a week for one hour sessions. The days of the week for
the groups are dependent upon the facilitator's schedule at that time. All youth in the program are
eligible to attend IOC groups and are required to successfully complete the curriculum before
graduating the program. Impact of Crime is held as a closed group, meaning no youth should be
added after the first lesson of chapter one.
The program is currently working on developing a binder to maintain all records for the IOC
cohorts. Currently, the sign-in sheets and fidelity monitoring forms are maintained in binders by
the facilitators of the group. Pursuant to a discussion with the clinical director, the binders will be
consolidated into one binder during the first quarter of the 2023-2024 review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An observation of Impact of Crime (IOC) was completed on Tuesday, December 13, 2022, from
2:10pm to 3:10pm. There were six youth, one direct care staff, and one facilitator attending group.
Two youth were off property and will receive a make-up lesson the following day.
The lesson for the day was a review of things that can interfere with decision making. The youth
completed role plays for Damien's Dillema. The youth were split into two grous of three, one
group conducted the Damien role-play and the second group came up with their own situaiton and
completed the role-play. The facilitator and youth had in-depth conversations surrounding the
role-plays and completed the discussion questions in the manual. The facilitator remained neutral
during the discussion and provided postive feedback. The youth were supportive of each other
throughout the process as well.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol in the event of Impact of Crime (IOC) facilitator turnover is for another
trained staff to continue with group facilitation. During the twelve (12)-month review period, the
program did experience turnover, with two facilitators leaving. One left during the third quarter and
one left during the fourth quarter. Both facilitators left during the cohorts they were facilitating.
Both cohorts had other trained staff continue with the group facilitation. There was one gap in
service due to facilitator turnover from June 21, 2022 to July 25, 2022 which was thirty-four (34)
days. The clinical director indicated this gap was due to having to wait for another staff to
complete their cohort in order to continue with the other cohort.
The sign-in sheets are maintained in the binders maintained by the facilitators of each cohort. All
sign-in sheets were provided for review. The program completed three cohorts during the review
period. The first cohort started October 14, 2021 and ended on February 15, 2022. The group
began with eight youth and seven youth successfully completed the curriculum. One youth was
transferred to another program due to behavioral issues. There was an eighteen (18)-day gap in
service during the cohort due to staff and youth having to quarantine. The missed sessions were
made-up as soon as everyone received an all-clear.
The second cohort started February 23, 2022 and ended on June 20, 2022. The cohort began
with eight youth and all eight youth successfully completed the curriculum. Groups were
consistently held two days a week for one hour. There were sign-in sheets included for the group
community service project.
The third cohort started May 23, 2022 and ended on October 27, 2022. The group began with ten
(10) youth and all ten (10) youth successfully completed the curriculum. This was the cohort with
the thirty-four (34)-day gap in service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
A fourth cohort began on September 13, 2022 and was continuing to be held at the time of this
review. The group began with eight youth and eight youth were still in the group at the time of the
review. This original facilitator has left employment; however, another staff was able to
immediately begin facilitation. This group is being held consistently two days a week for one hour.
A fifth cohort began on October 25, 2022 and was continuing to be held at the time of this review.
This group began with eight youth and eight youth continue with the group. This group is being
held two days a week for one hour.
The sign-in sheets for the IOC groups are not consistently being completed as required. This
reviewer spoke with the clinical director about the issues. The clinical director will be addressing
the issues with the facilitators. This reviewer will also be meeting with the facilitators during the
next review and discussing proper completion of the sign-in sheets.
The program can earn 2 points if facilitator turnover does not impact groups and there are no
unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Trained staff conduct Impact of Crime internal fidelity observations on each other every month.
The internal fidelity forms are located in the binders maintained by the facilitators of each cohort.
All forms were provided for review. During the twelve (12)-month review period, all internal fidelity
observations were completed for the six facilitators based upon when each facilitator conducted
groups.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any significant issues or corrective actions with the facilitators for
Impact of Crime (IOC) groups. All internal fidelity forms were reviewed for any issues and none
were noted. This reviewer also spoke with the clinical director and there were no corrective
actions for the facilitators of IOC.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
There were six facilitators during the twelve (12)-month review period. Of the six facilitators, two
were eligible for their annual performance evaluation. The performance evaluations include a
specific section to document what curricula the employee facilitates and how the employee is
doing with facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 17 youth in the
sample, 82% (14 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 17 youth in the
sample, 82% (14 of 17) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 10 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 65%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 3 youth
Moderate-High = 4 youth
High = 4 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 24%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 4 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 65% (11 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 24% (4 of 17) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 10 25 40%
Totals 66 85 78%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment.
17 youth completed the program during the review period. Of those youth, all 17 were included in
the sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 17 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended weeks of service. Youth in the sample completed between 13
and 19 weeks of service, with an average of 19 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, only 15 included dosage in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended hours of service. Youth in the sample completed between 12
and 46 hours of service, with an average of 35 hours.
The program was awarded 10 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 95%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can optimize their SPEP Quality of Service Delivery score
by ensuring facilitator turnover does not impact groups and there are no unnecessary gaps in
service.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 JUST IOC.pdf

service_type: Impact of Crime


summary: SPEP - FY22-23

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: November 1, 2021 - October 31, 2022
SPEP Review Conducted: December 13, 2022
Florida Department of Juvenile Justice
Report Date: 2/20/23
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Skillstreaming the Adolescent (Skillstreaming) as a primary service. The
therapy staff are trained to facilitate Skillstreaming groups. Facilitators receive an initial three-hour
training for Skillstreaming, with additional refresher training provided during clinical staff meetings.
Additionally, all program staff receive a one-hour basic skills training to orient everyone with the
material youth receive during group. This is done as a best practice to engage youth with skill
building outside of the classroom.
Eleven (11) staff were trained to facilitate Skillstreaming during the twelve (12)-month review
period. The program also utilizes internships through local schools, and four interns were trained
during the review period. Six of the eleven (11) staff have facilitated Skillstreaming groups during
the review period. All training records were provided for review. All eleven (11) staff were provided
appropriate training by the clinical director.
The clinical director conducts internal fidelity monitoring for the facilitators of Skillstreaming.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual was reviewed during the first quarter of the review period. The program uses the
2019 Skillstreaming the Adolescent manual, which includes separate youth and facilitator
manuals and skill cards.
Skillstreaming is required to be held two days a week for one hour. Between March 1, 2022 and
June 25, 2022 the groups were held four to five days a week. After June 25th, the groups were
held three days a week on a consistent basis. All youth are eligible to receive the Skillstreaming
curriculum. Skillstreaming is held as an open group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
All documentation for Skillstreaming groups are kept in a binder maintained in the clinical
director's office.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed and last approximately one hour.
Present for group were four youth, two facilitators, and the Clinical Director. The lesson for the
day was Sharing Something. The facilitator introduced the topic, had each youth write the skills on
the board, then journal what those skills mean to them. The group discussed how to work through
each step and then broke into two groups to model the learned skills. The group concluded by
working on a teamwork skill carried over from the previous session. The facilitators were
knowledgeable of the material but kept the manual close for reference to ensure they covered the
necessary topics; both adhered to the lesson plan.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's protocol, in the event of facilitator turnover for Skillstreaming the Adolescent
(Skillstreaming), is for another trained staff to continue with the groups. The therapists rotate
facilitation so the youth are familiar with the various facilitators.
The program did not experience facilitator turnover during the twelve (12)-month review period. A
review of sign-in sheets confirmed there were no gaps in service. Sign-in sheets were completed
correctly. Skillstreaming is held as an open group. The youth can begin the curriculum upon
admission to the program and participate throughout their stay in the program.
Skillstreaming groups were initially held four to five days a week between March 1, 2022 to June
25, 2022. Starting in July the grousp were held consistently three days a week.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director conducts internal fidelity observations for Skillstreaming the Adolescent
(Skillstreaming) groups.
Internal fidelity forms are maintained in a binder kept in the clinical director's office. If the clinical
director is facilitating a group, another trained therapist conducts fidelity monitoring. Fourteen (14)
out of fifty-four (54) observations were missing for the annual review period. After the second
quarter review, the internal fidelity observations were conducted on a more consistent basis.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any issues or corrective actions for facilitators of Skillstreaming the
Adolescent. The internal fidelity forms include a section to document any corrective actions for the
facilitators. A review of internal fidelity forms confirmed there were no corrective actions noted nor
taken during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program provided the annual performance evaluations for review. All of the evaluations
included a section which documented what curriculua the staff member was trained in during the
annual review period.
There were three annual performance evaluations completed. All of them were evaluated for their
group facilitation. Two other staff had not been at the program long enough to receive an annual
performance evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 14 youth in the
sample, 100% (14 of 14) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 14 youth in the
sample, 100% (14 of 14) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 10 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 40%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 64%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 2 youth
Moderate-High = 4 youth
High = 3 youth
Total Youth in Sample = 14 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 21%
High Score: 5 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 38%
Table 2
High = 3 youth
Total Youth in Sample = 14 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 64% (9 of 14) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 21% (3 of 14) youth scored High-Risk to reoffend, for a score of 5 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 10 25 40%
Totals 70 85 82%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Juvenile
Unit for Specialized Treatment.
17 youth completed the program during the review period. Of those youth, 14 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 14 total youth sampled, all
14 included dosage with end dates in the EBS Module. Of those youth, 14 received at least the
recommended weeks of service. Youth in the sample completed between 26 and 54 weeks of
service, with an average of 41 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 14 total youth
sampled, all 14 included dosage in the EBS Module. Of those youth, 14 received at least the
recommended hours of service. Youth in the sample completed between 50 and 57 hours of
service, with an average of 52 hours.
The program was awarded 10 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment can optimize their SPEP Quality of Service Delivery score
by ensuring fidelity monitoring is completed every month for each facilitator.
Juvenile Unit for Specialized Treatment can optimize their SPEP Amount of Service score by
ensuring dosage for all youth is recorded accurately in EBS and by ensuring youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2022-23 JUST Skillstreaming.pdf

service_type: Skillstreaming the Adolescent


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment (JUST)
Twin Oaks Juvenile Development, Inc
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Impact of Crime
SPEP Review Period: November 1, 2022 - October 31, 2023
SPEP Review Conducted: December 5, 2023
Florida Department of Juvenile Justice
Report Date: 6/26/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program uses Impact of Crime (IOC) as a primary service for all youth. Case management
staff are trained to facilitate IOC groups. There is a training protocol in place, delivered by a
qualified trainer. During the review period, there were four staff trained to facilitate IOC: three case
managers and one transition specialist. One case manager is no longer employed at the program.
There are two primary facilitators; other trained case managers conducted monthly internal fidelity
monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the current version of the Impact of Crime (IOC) manual. The manual includes
detailed lesson plans, a script for the facilitator, facilitator notes, and implementation information.
The manual also includes IOC workbooks for the youth. All facilitators have the newest version of
the manuals.
IOC groups are typically held on Tuesdays and Thursdays. All youth in the program are eligible to
attend IOC groups and are required to successfully complete the curriculum before graduating the
program. Impact of Crime groups are closed groups, meaning no youth should be added after the
first lesson of chapter one.
All records for the IOC cohorts are maintained in one binder, organized by facilitator/cohort.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Programming and technical assistance specialist observed a session of Impact of Crime on
December 14, 2023. Nine youth, the facilitator, two direct care staff, and the clinical director were
present. The session was a final recap and graduation celebration. The youth took turns
presenting key concepts and providing examples of the concepts. All youth were engaged and
excited about successfully completing the service. The facilitator and direct care staff were
encouraging and supportive.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Impact of Crime (IOC) groups is for
another trained staff to continue with the groups. All sign-in sheets were provided for review.
There were no gaps in services.
Four cohorts of IOC were facilitated during the review period. The first cohort was completed
during the first quarter. The cohort began September 13, 2022 and ended on January 5, 2023.
The group started with eight youth; all eight youth successfully completed the group. The group
was held consistently two days a week for one hour each session. One week the group had three
sessions instead of the usual two sessions per week. This group did experience facilitator
turnover; however, this information was captured in the previous annual report.
The second cohort began on October 25, 2022 with eight youth. The group ended on February
14, 2023, with all eight youth.
The third cohort began on March 14, 2023, ended on June 28, 2023. The group started with five
youth and three youth successfully completed the group.
The fourth cohort was started on July 11, 2023, and ended on October 3, 2023, with three youth.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct monthly internal fidelity monitoring. The trained
Impact of Crime (IOC) facilitators monitor each other, or the transition specialist conducts internal
fidelity monitoring observations monthly. There were two primary facilitators during the review
period.
All internal fidelity forms were provided for review. There were no missing internal fidelity forms
missing for the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring. The clinical director addresses any significant issues with the
facilitators for Impact of Crime (IOC). A review of the internal fidelity forms confirmed no
significant issues or corrective actions were noted nor taken.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The performance evaluations include a specific section to document what curricula the employee
facilitates and how the employee is doing with facilitation, including Impact of Crime. All eligible
faciltiators received an annual evaluation included feedback on their skill in delivering the service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 17 youth
High = 5 youth
Total Youth in Sample = 30 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 17%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 5 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (26 of 30) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 17% (5 of 30) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 13 25 52%
Totals 73 85 86%
This SPEP report evaluates Impact of Crime, an intervention delivered at Juvenile Unit for
Specialized Treatment (JUST).
30 youth completed the program during the review period. Of those youth, 30 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 30 total youth sampled, all
30 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 30
received at least the recommended weeks of service. Youth in the sample completed between 16
and 23 weeks of service, with an average of 19 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 30 total youth
sampled, all 30 included dosage in the EBS Module. Of those youth with correct dosage, 30
received at least the recommended hours of service. Youth in the sample completed between 31
and 35 hours of service, with an average of 32 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment (JUST) can maintain their SPEP Quality of Service
Delivery score by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment (JUST) can maintain their SPEP Amount of Service score
by continuing to ensure dosage for all youth is recorded accurately in EBS and by continuing to
ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 JUST IOC.pdf


summary: SPEP - FY23-24

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Juvenile Unit for Specialized Treatment (JUST)
Twin Oaks Juvenile Development, Inc
(Contract Provider)
29841 Liberty Wilderness Road
Sumatra, Florida 32335
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: November 1, 2022 - October 31, 2023
SPEP Review Conducted: December 5, 2023
Florida Department of Juvenile Justice
Report Date: 6/26/2024
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program uses Skillstreaming the Adolescent (Skillstreaming) as a primary service. The
therapy staff are trained to facilitate Skillstreaming groups. Facilitators receive an initial three-hour
training, with additional refresher training provided during clinical staff meetings. Additionally, all
program staff receive a one-hour basic skills training to orient everyone with the material youth
receive during group. This is done as a best practice to engage youth with skill building outside of
the classroom.
During the review period, five staff were trained in the Skillstreaming curriculum: one clinical
director and five clinical coordinators. The staff rotate group facilitation, so all five staff facilitated
Skillstreaming groups, at least once, during the review period. Two of the five therapists are no
longer employed at the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program uses the 2019 Skillstreaming the Adolescent manual, which includes separate youth
and facilitator manuals and skill cards. Skillstreaming is held at least three days a week, typically
Tuesday, Thursday, Friday, and/or Saturday. All youth are required to participate in the service.
Skillstreaming is held as an open group, meaning you may be added to the service upon their
arrival at the program, without a need to wait for a new group to form.
All documentation for Skillstreaming groups is kept in a binder maintained in the clinical director's
office.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Programming and technical assistance specialist observed a session of Skillstreaming the
Adolescent on December 20, 2023. The session began at 2:35 and lasted approximated 45
minutes. Present for group were eight youth, one direct care staff, and the facilitator. The topic of
discussion was starting a conversation, which led into the following skill- having a conversation.
The youth were given the skill, asked to explain the meaning in their own words, and followed up
with a role play exercise to model the skill. Several youths were not engaged in the discussion
and the facilitator used motivational interviewing skills to encourage their participation. The youth
were displeased with having to go to a second group (Boys Council) immediately following this
session. This reviewer discussed the complaint with the Clinical Director, who indicated they are
working on a schedule change to alleviate this issue.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy in the event of facilitator turnover for Skillstreaming groups is for another
trained staff to continue with group facilitation. The clinical coordinators rotate facilitation, so the
youth are familiar with all trained staff. Skillstreaming is held as an open group, run year-round.
All sign-in sheets were provided for review. There were no gaps in service.
There are typically three groups held during each week. with four different staff facilitating groups,
including Saturdays. Two facilitators left employment; however, the groups were held consistently
during this review period. There were between five and ten (10) youth in the groups.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process in place to conduct monthly internal fidelity monitoring. The process is
for the clinical director or the other trained staff to conduct monthly observations for all facilitators.
All internal fidelity observation forms were provided for review.
There were two fidelity observations missing for two of the facilitators during the first quarter of the
review period. This reviewer spoke with the clinical director regarding the missing observations.
The clinical director indicated the two facilitators co-facilitated the Skillstreaming groups, but only
one facilitator signed the sign-in sheets. For the other missing fidelity form, the observer dated the
form one day before the actual observation occurred.
This discrepancy was discussed with the clinical director. The fidelity forms must be dated on the
date the actual observation was conducted. This observation will not be counted for the review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions identified during monthly
internal fidelity monitoring. The clinical director addresses any issues or corrective actions for
facilitators of Skillstreaming the Adolescent. The internal fidelity forms include a section to
document any corrective actions for the facilitators.
A review of internal fidelity forms confirmed there were no corrective actions noted nor taken
during the annual review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Employee evaluations include a section to document which curricula the staff is trained to deliver.
Each trained staff eligible for an annual performance evaluation was evaluated on their skill in
delivering Skillstreaming the Adolescent.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 16 weeks for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 24 contact hours for this type of service. Of the 30 youth in the
sample, 100% (30 of 30) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 13 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 52%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 87%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 4 youth
Moderate-High = 17 youth
High = 5 youth
Total Youth in Sample = 30 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 17%
High Score: 3 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 23%
Table 2
High = 5 youth
Total Youth in Sample = 30 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 87% (26 of 30) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 17% (5 of 30) youth scored High-Risk to reoffend, for a score of 3 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 13 25 52%
Totals 73 85 86%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at Juvenile
Unit for Specialized Treatment (JUST).
30 youth completed the program during the review period. Of those youth, 30 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 10 points for Amount of Service: Duration. Of the 30 total youth sampled, all
30 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 30
received at least the recommended weeks of service. Youth in the sample completed between 18
and 61 weeks of service, with an average of 38 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 10 points for Amount of Service: Contact Hours. Of the 30 total youth
sampled, all 30 included dosage in the EBS Module. Of those youth with correct dosage, 30
received at least the recommended hours of service. Youth in the sample completed between 50
and 52 hours of service, with an average of 51 hours.
The program was awarded 13 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 100%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Juvenile Unit for Specialized Treatment (JUST) can maintain their SPEP Quality of Service
Delivery score by continuing the practices in place at the time of this review.
Juvenile Unit for Specialized Treatment (JUST) can maintain their SPEP Amount of Service score
by continuing to ensure dosage for all youth is recorded accurately in EBS and by continuing to
ensure youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2023-24 JUST Skillstreaming.pdf