summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Duval Academy
Sequel Youth and Family Services
(Contract Provider)
7500 Ricker Rd
Jacksonville, Florida 32244
Primary Service: Thinking for a Change
SPEP Review Period: April 15, 2021-April 14, 2022
SPEP Review Conducted: May 25, 2022
Florida Department of Juvenile Justice
Report Date: 8/3/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed the program utilizes Thinking for a Change (T4C) as a primary service. The
documentation shows there were three staff trained in the primary service: two case managers
and the transition specialist. T4C training is a four days, conducted by a qualified trainer. All three
trained staff facilitated the primary service during the annual review period. Any trained staff
member may conduct monthly internal fidelity checks. The transition specialist, one case
manager, and a trained staff from another program conducted the monthly fidelity checks during
the annual review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The documentation reviewed for the annual review period of April 15, 2021 through April 14, 2022
shows the program utilized the Thinking for a Change 4.0 manual. The manual includes twenty-
five (25) lessons. Each lesson has a script for the facilitator and instructions for each activity. The
manual requires the lessons be given in order and provides specific instructions for the
implementation of the program. There are no separate student workbooks. The facilitators make
copies of the current lessons to hand out to the youth.
T4C is delivered as a closed group with no more than twelve (12) youth in each cohort. The
curriculum is to be delivered to the youth over a minimum of fifteen (15) weeks for forty-five (45)
hours. The program keeps all SPEP documents in a binder labeled "Thinking for a Change" in the
clinical director’s office. The program provides the service on Wednesdays and Fridays for one-
and one-half hour each. All youth receive Thinking for a Change.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change (T4C) was observed on 5/25/2022 via Microsoft Teams. There
were twelve (12) youth and the facilitator present for group. The group began at 3:17pm and was
held in the multi-purpose room. The youth were being disrespectful during the check in and
throughout the session. The youth were having side conversations stating the executive director
was trying to keep them in the program as long as he could. The youth stated they had targets on
their backs because of the executive director. The facilitator attempted to follow the lesson, but
the youth kept going back to their grievance. The facilitator would read a section and then ask the
youth a question. The youth would not answer the question. They would reengage with their side
conversation. The facilitator did not have any visual aids available. The facilitator did not have the
youth complete all the activities during the lesson. The group ended at 4:45p.
The technical assistance specialist spoke with the clinical director and the facilitator about the
youth being disrespectful, not having visual aids, and not completing all exercises/activities during
each session. The facilitator stated he did not prepare enough in advance as he would have liked.
The technical assistance specialist stated there were videos and PowerPoints associated with
each lesson. The facilitator did not know if he had access to them. The technical assistance
specialist also informed the facilitator he was missing completing all exercises during the lessons.
The facilitator stated he did not realize he missed those activities. The facilitator stated he handed
the youth their homework sheets and thought that was the only activity (other than role plays) the
youth needed to complete. The technical assistance specialist stated the youth need to complete
all activities within the lesson and the facilitator should explain the homework instead of just
handing the youth the worksheets. The technical assistance specialist spoke with the facilitator
about the role plays. The facilitator did not model the role plays as specified in the manual. The
facilitator jumped straight to the youth role plays. The facilitator did not realize he missed a step.
The program can earn 1 point if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of April 15, 2021 showed the program
did have facilitator turnover which adversely affected the delivery of Thinking for a Change. A
case manager left the program in June 2021 and another case manager left the program at the
end of October 2021. The case manager who left at the end of October 2021 was facilitating a
cohort. Another trained facilitator took over for the last three sessions.
Documentation reviewed shows there were four cohorts completed and one cohort ongoing at the
time of the annual review. There were three trained staff members (two case managers and the
transition specialist) who facilitated all five cohorts during the annual review period. The dates of
the cohorts are 2/12/2021-5/26/2021, 6/22/2021-10/25/2021, 10/27/2021-2/7/2022, 2/17/2022-
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
4/28/2022, and 2/18/2022-current. There was a gap of twenty-seven (27) days between the cohort
which ended on 5/26/2021 and the cohort which began on 6/22/2021. The gap in service was due
to the program’s low census and no new admissions until July 2021 and does not qualify as an
unnecessary gap in service.
The 2/12/2021 cohort began with seven youth and ended with seven youth.
The 6/22/2021 cohort began with three youth and ended with three youth. One youth was added
to the cohort on 7/7/2021. The youth was admitted to the program shortly after the cohort began.
The facilitator conducted one-on-one supportive sessions with the new youth on 7/3/2021 and
7/4/2021 prior to the youth being added to the cohort. One youth left the program on 10/13/2021.
The case manager who began the cohort facilitated all lessons except for the last three. The
former clinical director stated the youth only had forty-one (41) and a half hours at the time the
case manager left. The clinical director said she would have the transition specialist conduct the
final three sessions to ensure the youth received a minimum of forty-five (45) hours. The transition
specialist took over on 10/20/2021.
The 10/27/2021 cohort began with twelve (12) youth and ended with ten (10) youth. One youth
was arrested on 11/2/2021. One youth took HUB on 1/24/2022 and was released on 1/26/2022.
The 2/17/2022 cohort began with six youth and currently has five youth. One youth left the cohort
between 3/30/2022 and 4/3/2022. The attendance sheet did not document the date nor reason
the youth left the cohort. The 2/18/2022 cohort began with twelve (12) youth and currently has
twelve (12) youth.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of April 15, 2021 through April 14, 2022
showed there were ten (10) internal fidelity checks completed for facilitators of Thinking for a
Change. Three staff facilitated groups during the annual review period. The program was a
missing fidelity check for the month of June 2021 for a case manager. The former clinical director
stated the missing fidelity was due to the cohort beginning close to the end of the month. The
program was missing a fidelity check for the month of September 2021 for the case manager. The
program was missing a fidelity check for the months of October 2021 and April 2022 for the
transition specialist. A case manager, the transition specialist, and another trained staff from
another program conducted the internal fidelity monitoring during the annual review period. The
clinical director stated the program conducts internal fidelity monitoring on a monthly basis.
Facilitators trained must be monitored monthly by another trained and experienced facilitator.
The transition specialist used a fidelity monitoring form with preprinted marks for the fidelity
checks dated for 10/13/2021 and 8/4/2021. The fidelity monitoring showed the transition specialist
had the "yes" box marked for behavioral issues, but the narrative section stated there were no
behavioral issues noted. The technical assistance specialist informed the executive director,
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
former clinical director, and regional clinical director the fidelity form boxes should not be pre-
populated. They should be filled out during the actual fidelity check. There were three fidelity
checklists (1/26/2022, 2/24/2022, and 3/25/2022) which had N/A listed as the facilitator, another
trained staff member from another program as the co-facilitator, and the transition specialist as
the observer. The clinical director stated the trained staff from another program came over to
conduct the fidelity check and the transition specialist was the facilitator. The clinical director filled
out and signed a fidelity checklist on 4/28/2022 without being trained in the service. The clinical
director stated she completed the fidelity check because she found out late the other trained staff
member was not coming over to the program during the month of April 2022. Technical
assistance specialist stated the fidelity check completed by the clinical director would not count
since she was not trained in the primary service.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period showed there were no identified
corrective actions noted during the monthly internal fidelity checks for facilitators of Thinking for a
Change. The clinical director stated when corrective action is needed, the process will be for the
facilitator to meet with the clinical director to review findings from the completed fidelity monitoring
to address concerns and provide support. The facilitator will be re-assessed at the next month’s
fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The documentation reviewed for the annual review period of April 15, 2021 through April 14, 2022
shows each evaluation has an area called "facilitator skill delivering the evidence-based
programming". Two evaluations were provided for staff who facilitated Thinking for a Change. The
transition specialist’s evaluation identified the specific SPEP services he was facilitating. The
evaluation mentioned all of your "SPEP groups are on target to meet the specific dosage," but did
not contain any other specific information. A case manager’s evaluation did contain specific
information about the facilitator’s skill in delivering the primary service.
The program can earn 1 point if facilitator’s performance evaluation includes discussion of their
skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 13 youth in the
sample, 85% (11 of 13) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 45 contact hours for this type of service. Of the 13 youth in the
sample, 85% (11 of 13) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 100%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 5 youth
Total Youth in Sample = 13 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 38%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 5 youth
Total Youth in Sample = 13 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 100% (13 of 13) youth scored Moderate to High-Risk to reoffend, for a
score of 12 points.
Of the SPEP sample, 38% (5 of 13) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 86 100 86%
This SPEP report evaluates Thinking for a Change, an intervention delivered at Duval Academy.
13 youth completed the program during the review period. Of those youth, all 13 were included in
the sample.
The program scored Medium for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 13 total youth sampled, all
13 included dosage with end dates in the EBS Module. Of those youth, 11 received at least the
recommended weeks of service. Youth in the sample completed between 8 and 15 weeks of
service, with an average of 14 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 13 total youth
sampled, all 13 included dosage in the EBS Module. Of those youth, 11 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 17 and 45 hours of
service, with an average of 43 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
Duval Academy can optimize their SPEP Quality of Service Delivery score by ensuring the
facilitator adheres to the manual, facilitator turnover does not impact groups, fidelity monitoring is
completed every month for each facilitator, and facilitator’s performance evaluation includes
discussion of their skill in delivering this primary service.
Duval Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 Duval T4C.pdf