summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Impact of Crime
SPEP Review Period: 2/1/ 2020-1/31/2021
SPEP Review Conducted: 3/22/2021
Florida Department of Juvenile Justice
Report Date: 4/14/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Impact of Crime as a primary service for the annual review period of
February 2020 through January 2021. A review of the program's documentation from the twelve-
month review period showed five staff were trained to conduct Impact of Crime groups. The
trained staff are three case managers, the transition manager, and the clinical director. The
training is a three-day training conducted by a qualified trainer. The documentation shows the
case manager and the transition manager conduct the monthly internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm v 2.0 as the manual. The
facilitator’s manual is divided into seven chapters, an appendix with handouts, and the student
guide section (which includes the student exercises). The manual addresses the order in which to
deliver the content and gives the facilitators an outline to follow. The facility does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth.
The group is a closed group with no more than twelve youth in each cohort. The curriculum is to
be delivered to the youth over a minimum of sixteen weeks for thirty-two sessions. The
documentation shows the primary service is delievered on Wednesdays 4p-5:30p and Saturdays
11:00a-12p.
The program keeps all sign in sheets, internal fidelity checks, and training certificates in a binder
titled IOC SPEP binder.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 1/13/2021. The facilitator began group by having
youth read exercise number nine. Youth were divided into two groups, to come up with effects of
drugs on a pregnant woman. The group that comes up with the most will win a prize. The
facilitator wrote four questions on the board and told the youth they would research the answers
to those questions. The facilitator then stated they have reached the end of chapter four and
would begin chapter five next group. The facilitator asked the youth to define aggravated assault,
gang, robbery, ripple effect, and sexual battery. The facilitator then went over the chapter five
objectives. The youth participated in an experiment with baking soda and vinegar which
demonstrated a cause and effect relationship. The facilitator asked the youth about the four victim
impacts and stinking thinking. The youth then read the Calvin and Jaden scenarioand were asked
to identify the positive and negative effects from the scenario.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program did have facilitator turnover; however, the
turnover did not affect the groups, as the facilitator who left was not facilitating groups at that time.
There were four cohorts completed during the annual review period. The dates of the cohorts are
1/20/2020-5/5/2020, 5/20/2020-10/3/2020, 11/11/2020-current, and 8/29/2020-12/20/2020. There
were missing sign in sheets for the cohort beginning 11/11/2020. The last sign in sheet for the
cohort which began on 11/11/2020 was dated for 12/12/2020.
The 1/20/2020 cohort began with ten youth. The documentation showed there was one youth
discharged on 1/23/2020, but the youth was back on 1/27/2020. The same youth was again
discharged on 3/9/2020. The documentation also showed the facilitator skipped to chapter five for
the lesson on 1/23/2020, which was only three days after beginning the cohort.
The 5/20/2020 cohort began with eight youth. There were two youth released from the program in
August 2020. The director of case management noted on the attendance sheets for 7/21/2020
and 7/23/2020 there was no groups held due to the youth being in medical quarantine because of
the COVID-19 pandemic.
The11/11/2020 cohort has the last sign in sheet dated for 12/12/2020. There were eight youth
who began the group.
The 8/29/2020 cohort began with ten youth. There was one youth added to the cohort on
9/19/2020 per the note by the transition manger. There were sign in sheets dated in August 2020
and early September 2020 documenting homework being completed for chapters one and two for
the youth admitted to the cohort after it began. All youth in the cohort were released from the
program prior to successfully completing the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation showed the program completed twelve internal fidelity monitorings.
The program was missing internal fidelity monitoring for the months of May and September 2020,
and January 2021. The clinical director stated the process for internal fidelity monitoring is to have
the trained facilitators take turns completing fidelity checks on each other.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation shows there were no corrective action noted during the review period.
The corrective action process is to have monthly fidelity checks completed on each facilitator. If a
corrective action is noted during the fidelity, then he will discuss it with the facilitator and then re-
evaluate the corrective action during the next fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation shows the program has completed one annual evaluation for a trained
facilitator. The evaluation has an area titled facilitator skill delivering evidence-based practices.
The one evaluation had remarks which discussed the facilitator’s skill in delivering Impact of
Crime.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 49 youth in the
sample, 65% (32 of 49) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 49 youth in the
sample, 84% (41 of 49) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 80%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 3 youth
Moderate-High = 15 youth
High = 21 youth
Total Youth in Sample = 49 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 43%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 21 youth
Total Youth in Sample = 49 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 80% (39 of 49) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 43% (21 of 49) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 74 85 87%
This SPEP report evaluates Impact of Crime, an intervention delivered at St. Johns Youth
Academy.
60 youth completed the program during the review period. Of those youth, 49 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 49 total youth sampled, only
47 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 32
received at least the recommended weeks of service. Youth in the sample completed between 9
and 29 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 49 total youth
sampled, only 47 included dosage in the EBS Module. Of those youth with correct dosage, 41
received at least the recommended hours of service. Youth in the sample completed between 18
and 50 hours of service, with an average of 30 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
St. Johns Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator.
St. Johns Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: Saint Johns Youth Academy

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Skillstreaming the Adolescent
SPEP Review Period: 2/1/2020-1/31/ 2021
SPEP Review Conducted: 3/23/2021
Florida Department of Juvenile Justice
Report Date: 4/14/2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Skillstreaming the Adolescent as a primary service. A review of the
program's documentation showed nine staff were trained in the primary service. Eight of the nine
staff facilitated the primary service during the annual review period. Eight of the nine trained staff
are therapists and the ninth is the clinical director. The program has a written training protocol for
the Skillstreaming the Adolescent service. The training is a four-hour in-service training that is
conducted by the clinical director. The training is broken down into three parts. The first part is the
introduction to Skillstreaming the Adolescent. This section is one hour in length. The second
section is the review of materials. This section is two hours in length. The third section is the
observation by the clinical director of the trained staff conducting a group session. The clinical
director conducts the monthly internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Skillstreaming the Adolescent 3rd edition. The manual introduces
Skillstreaming, contains scripts for the facilitators to follow as well as specific instructions for each
skill taught, example role plays, and student exercises. There are no separate student manuals.
The facilitator makes copies of the skill cards for each skill and hands out those copies to the
youth in the group. There are fifty skills taught to the youth.
Skillstreaming is an open group with groups being run year-round. The program maintains the
attendance sheets and monthly internal fidelity checks in the SPEP binder titled Skillstreaming.
Skillstreaming is held on Tuesdays and Thursdays. All youth receive the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed on 1/12/2021. The group began at
2:35p. The facilitator began with group check in. The youth were asked to rate their day, name
their emotion, and then asked who was the first person who helped you out at the program and
what did they help you. The facilitator asked the youth to read one step each. The facilitator
assigned the youth to role play. One youth refused to participate with the group. The skill steps
were written on the white board. The facilitator finished skill twenty-four by 3:05p. The facilitator
decided to do a review of last Thursday’s skill: asking for permission. The youth then began
playing hangman. The facilitator reminded the youth to ask for help. The group ended at 3:25p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program had facilitator turnover; one therapist left the
program during the annual review period. The youth were placed with another therapist. The
groups are open groups ran year-round. The youth are assigned to a mod as well as a trained
facilitator. If there is facilitator turnover, the program will assign another trained facilitator to the
mod to continue the delivery of the primary service. There were no gaps in service.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the annual review period of February 2020 through January 2021
showed the program completed fifty-eight internal fidelity monitorings, accounting for each
facilitator each month. The clinical director completes all monthly internal fidelity checks.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation shows there were corrective actions noted during the review period.
There were eight fidelity checks which notated a corrective action needed. There were five
consecutive fidelities for one therapist which notated a need for additional monitoring. The clinical
director stated the therapist was continually in and out of the program due to issues out of their
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
control so the additional fidelities were unable to come to fruition. The clinical director did notate
the facilitator is improving, but could use additional assistance on each subsequent fidelity check.
The corrective action process is to conduct monthly fidelity checks on each facilitator. If a
corrective action is noted during the fidelity, then he will discuss it with the facilitator and then re-
evaluate the corrective action during the next fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation of the annual review period shows the program has completed six
annual evaluations for the trained facilitators. One of the trained facilitators was not employed at
the program for a full year prior to his departure. The evaluation has an area titled facilitator skill
delivering evidence-based practices. All evaluations included comments addressing their skill in
facilitating Skillstreaming the Adolescent.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 59 youth in the
sample, 97% (57 of 59) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 59 youth in the
sample, 97% (57 of 59) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 15 youth
High = 29 youth
Total Youth in Sample = 59 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 49%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 29 youth
Total Youth in Sample = 59 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (49 of 59) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 49% (29 of 59) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 76 85 89%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at St. Johns
Youth Academy.
60 youth completed the program during the review period. Of those youth, 59 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 59 total youth sampled, all
59 included dosage with end dates in the EBS Module. Of those youth, 57 received at least the
recommended weeks of service. Youth in the sample completed between 5 and 72 weeks of
service, with an average of 38 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 8 points for Amount of Service: Contact Hours. Of the 59 total youth
sampled, all 59 included dosage in the EBS Module. Of those youth, 57 received at least the
recommended hours of service. Youth in the sample completed between 18 and 72 hours of
service, with an average of 40 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 95%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
St. Johns Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups.
St. Johns Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Saint Johns Youth Academy

program_status_note: Closed

service_type: Skillstreaming the Adolescent


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Thinking for a Change
SPEP Review Period: 2/1/2020-1/31/2021
SPEP Review Conducted: 3/23/2021
Florida Department of Juvenile Justice
Report Date: 4/14/21
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Thinking for a Change (T4C) as a primary service. A review of the program's
documentation showed seven staff were trained in the primary service. The training is thirty-two-
hours and is completed by a qualified trainer. The documentation showed there were four staff
conducting groups during the review period. The four staff conducting the primary service were a
case manager, the recreational therapist, the assistant facility administrator, and an educational
specialist. There were four staff who conducted the monthly internal fidelity monitoring. Those
staff were all case managers.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Thinking for a Change 4.0 manual. The manual includes twenty-five
lessons. Each lesson has a script for the facilitator and instructions for each activity. The manual
requires the lessons be given in order and provides specific instructions on the implementation of
the program. There are no separate student workbooks. The facilitators make copies of the
current lessons to hand out to the youth.
The program maintains the attendance sheets and monthly internal fidelity checks in the SPEP
binder titled T4C. The program utilizes three different groups for Thinking for a Change. Two
groups are held on Tuesdays and Thursdays. One group is held on Mondays and Saturdays. All
youth are eligible to receive the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed on 3/22/2021. The group began at 4p. The
facilitator began with group check in. The facilitator began the lesson on giving feedback, going
over the skill steps for giving feedback. The facilitator was observed to read from the facilitator’s
manual. The facilitator reviewed the skill steps with the youth and asked them if the step was a
thinking step or an action step. The facilitator then modeled the skill. The youth were then asked
to role play the skill. The youth completed an activity after the role plays were concluded. The
group ended at 4:56p.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed shows the program did not have facilitator turnover. There were two
case managers who left the program during the annual review period, but they were not
facilitating groups. If there is facilitator turnover, the program will assign another trained facilitator.
There were no gaps in service.
There was documentation of six cohorts being run during the review period as evidenced by the
sign in sheets. The dates of the cohorts were 8/24/2020-11/20/2020, 9/1/2020-12/10/2020, (no
attendance sheet which showed the first session)-9/18/2020, (no attendance sheet which showed
the first session)-5/13/2020, 6/15/2020-10/10/2020, and 11/7/2020-current.
The documentation reviewed showed the cohort which began on 11/7/2020 did not have a
completion date. The last attendance sheet was dated 12/28/2020. There were no other
attendance sheets provided for this cohort. The cohort began with eight youth.
The cohort which began on 6/15/2020 began with seven youth and ended with six youth. One
youth was released prior to completing the primary service. There was a note on the attendance
sheets for 7/21/2020 and 7/23/2020 which stated no groups were held this week due to youth
being in quarantine.
The cohort which ended on 5/13/2020 did not have an attendance sheet documenting the first
lesson. The first attendance sheet for this cohort was dated on 4/13/2020.
The cohort which ended on 9/18/2020 did not have an attendance sheet documenting the first
lesson. The first attendance sheet for this cohort was dated on 8/19/2020.
The cohort which began on 9/1/2020 began with nine youth and ended with two youth. Seven
youth were released prior to completing the primary service.
The cohort which began on 8/24/2020 began with nine youth and ended with six. Three youth
were released prior to completing the primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the annual review period showed the program completed
seventeen internal fidelity monitorings, accounting for each facilitator each month. The internal
fidelity monitoring was completed by staff trained in the primary service.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of documentation shows there were no corrective actions noted during the review period.
The corrective action process is to complete monthly fidelity checks on each facilitator and if a
corrective action is noted during the fidelity, then he will discuss it with the facilitator and then re-
evaluate the corrective action during the next fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation shows the program has completed two annual evaluations for the
trained facilitators. The evaluation has an area titled facilitator skill delivering evidence-based
practices. The evaluations reviewed had remarks which discussed the facilitator’s skill in
delivering Thinking for a Change.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 15 weeks for this type of service. Of the 58 youth in the
sample, 55% (32 of 58) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 58 youth in the
sample, 43% (25 of 58) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 84%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 5 youth
Moderate-High = 17 youth
High = 27 youth
Total Youth in Sample = 58 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 47%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 27 youth
Total Youth in Sample = 58 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 84% (49 of 58) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 47% (27 of 58) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 20 25 80%
Totals 83 100 83%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St. Johns Youth
Academy.
60 youth completed the program during the review period. Of those youth, 58 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 58 total youth sampled, only
51 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 32
received at least the recommended weeks of service. Youth in the sample completed between 1
and 19 weeks of service, with an average of 14 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 58 total youth
sampled, only 51 included dosage in the EBS Module. Of those youth with correct dosage, 25
received at least the recommended hours of service. Youth in the sample completed between 3
and 50 hours of service, with an average of 39 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 88%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
St. Johns Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service.
St. Johns Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Thinking for a Change
SPEP Review Period: 2/1/2021-1/31/2022
SPEP Review Conducted: 3/17/2022
Florida Department of Juvenile Justice
Report Date: 4/12/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Thinking for a Change (T4C) as a primary service for the annual review
period of February 1, 2021 through January 31, 2022. A review of documentation for the annual
review period showed five staff were trained in T4C- two assistant facility administrators, one
recreation therapist, and two case managers. The training is a thirty-two-hour training completed
by a qualified trainer. The documentation showed there were two case managers who conducted
groups during the annual review period. There were three staff who conducted the monthly
internal fidelity monitoring. The three staff who conducted the monthly internal fidelity monitoring
were the two assistant facility administrators and the recreation therapist. Any trained staff
member can conduct the monthly fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Thinking for a Change 4.0 manual. The manual includes twenty-five
lessons. Each lesson has a script for the facilitator and instructions for each activity. The manual
requires the lessons be given in order and provides specific instructions on the implementation of
the program. There are no separate student workbooks. The facilitators make copies of the
current lessons to hand out to the youth. The program maintains the attendance sheets and
monthly internal fidelity checks in the SPEP binder titled "T4C". The program states all youth are
to receive the primary service. The current clinical director states the program decides which
youth receives the intervention first by determining if the youth has previously participated in the
completion of the required dosage of services. If the youth has not, the youth is automatically
included in the next incoming cohort. If the youth has completed the required dosage, but
continues to struggle with personal accountability for their actions, they can be re-admitted in the
next incoming cohort.
The primary service is a closed group, held on Saturdays, Mondays, and Wednesdays.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed on 3/17/2022. The group began at 3:02pm with
a group check in. The facilitator reviewed the prior lesson before moving on to the current lesson
“Apologizing.” The facilitator had two blank pieces of chart paper on the wall. The facilitator did not
have the lesson content already written out. The facilitator read the information from the manual
verbatim, but was able to fully explain the content to the youth to ensure they understood the
concepts presented. The facilitator was knowledgeable of the material. The group ended at 3:55p.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of February 1, 2021 through January
31, 2022 shows the program had facilitator turnover which affected delivery of Thinking for a
Change. The program conducted four cohorts during the annual review period. The dates of the
cohorts were 11/7/2020-2/13/2021, 3/13/2021-6/14/2021, 3/16/2021-4/22/2021, and 7/24/2021-
10/28/2021. There was a gap of one month between the cohort which ended on 2/13/2021 and
the cohort which began on 3/13/2021. There was another gap of forty days between the cohort
which ended on 6/14/2021 and the cohort which began on 7/24/2021. The program did not have a
reason for the two gaps in service. There was another gap in service between the cohort which
ended on 10/28/2021 and the cohort which began in February 2022.
The 11/7/2020 cohort began with eight youth and ended with six youth. This cohort began during
the last review period and ran into the current annual review.
The 3/16/2021 cohort began with eight youth and ended with zero youth completing the service.
The last attendance sheet was dated on 4/22/2021. The facility administrator stated all the youth
were discharged prior to completing the service. The attendance sheet dated for 4/6/2021 did not
have the lesson marked.
The 3/13/2021 cohort began with nine youth and ended with eight youth.
The 7/25/2021 cohort began with seven youth and ended with seven youth.
The facility administrator stated the process, in the event of facilitator turnover, is the program
plans to implement a co-facilitator guideline in which there are two facilitators running each group,
so in the event one of the facilitators is out for any reason, the group can continue.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed shows the program completed twelve internal fidelity monitorings for
facilitators of Thinking for a Change, accounting for each month groups were held. The program is
missing monthly fidelity monitoring for the months of November 2021, December 2021, and
January 2021 due to the program not running the primary service. The monthly fidelity
monitorings were completed by the two assistant facility administrators and the recreation
therapist. The facility administrator stated the program conducts internal fidelity monitoring
monthly by another trained and experienced facilitator.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period of February 1, 2021 through January
31, 2022 shows the program did not identify any corrective actions on the monthly internal fidelity
monitorings for facilitators of Thinking for a Change. The facility administrator stated the
program’s in-house corrective action process is, when corrective action is needed, the facilitator
will meet with the clinical director to review findings from the completed fidelity monitoring to
address concerns and provide support. The facilitator will be re-assessed at the next month’s
fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation of the annual review period shows the program provided an updated
annual evaluation for the recreation therapist. The evaluation has an area titled facilitator skill
delivering evidence-based practices. The evaluation had remarks which specifically discussed the
facilitator’s skill in delivering Thinking for a Change.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 15 weeks for this type of service. Of the 49 youth in the
sample, 8% (4 of 49) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 49 youth in the
sample, 18% (9 of 49) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 88%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 19 youth
High = 23 youth
Total Youth in Sample = 49 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 47%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 23 youth
Total Youth in Sample = 49 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 88% (43 of 49) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 47% (23 of 49) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 78 100 78%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St. Johns Youth
Academy.
56 youth completed the program during the review period. Of those youth, 49 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 49 total youth sampled, only
11 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 4
received at least the recommended weeks of service. Youth in the sample completed between 1
and 17 weeks of service, with an average of 7 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 0 points for Amount of Service: Contact Hours. Of the 49 total youth
sampled, only 36 included dosage in the EBS Module. Of those youth with correct dosage, 9
received at least the recommended hours of service. Youth in the sample completed between 1.5
and 45 hours of service, with an average of 27 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
St. Johns Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service.
St. Johns Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 SJYA T4C.pdf


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Impact of Crime
SPEP Review Period: 2/1/2021-1/31/2022
SPEP Review Conducted: 3/17/2022
Florida Department of Juvenile Justice
Report Date: 4/12/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Impact of Crime (IOC) as a primary service for the annual review period of
February 1, 2021 through January 31, 2022. A review of documentation showed five staff were
trained in IOC- one assistant facility administrator, one recreation therapist, one transition
manager, the director of case management, and one case manager. The training is a three-day,
24-hour training currently being provided online via Microsoft Teams. The documentation showed
there one case manager and one recreation therapist conducted groups. Four staff conducted
monthly internal fidelity monitoring. The four staff who conducted the monthly internal fidelity
monitoring were the director of case management, one case manager, the transition manager,
and the recreation therapist. Any trained staff member can conduct monthly fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the curriculum Impact of Crime: Addressing the Harm v 2.0. The facilitator’s
manual is divided into seven chapters, an appendix with handouts, and the student guide section
(which includes the student exercises). The manual addresses the order in which to deliver the
content and gives the facilitators an outline to follow. The facility does not have separate student
workbooks to give out. The facilitators make copies of the exercises before each session to pass
out to the youth.
The group is a closed group with no more than twelve youth in each cohort. The curriculum is to
be delivered to the youth over a minimum of sixteen weeks for thirty-two sessions. The
documentation shows Impact of Crime (IOC) is delivered on Wednesdays 4p-5:30p and
Saturdays 11:00a-12p. The program keeps all sign in sheets, internal fidelity checks, and training
certificates in a binder titled IOC SPEP binder. All youth in the program receive IOC. The current
clinical director states the program decides which youth receives the intervention first by
determining if the youth has previously participated in the completion of the required dosage of
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
services. If the youth has not, the youth is automatically included in the next incoming cohort. If
the youth has completed the required dosage, but continues to struggle with personal
accountability for their actions, they can be re-admitted in the next incoming cohort.
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was observed on 3/7/2022 via Microsoft Teams. The group began at
5:06p. The facilitator stated the group was going to review chapters one and two. The group was
working on the end of chapter three. The facilitator stated he allows the youth to lead different
groups while the facilitator ensures they are going over the correct information. The facilitator will
ask clarifying questions to ensure the youth understand the concepts before moving to a different
lesson. A youth led the group during the final exercise for chapter three. The facilitator was
knowledgeable of the material. The group ended at 5:32p.
The technical assistance specialist informed the facilitator and the clinical director the youth
should not be leading the group. The facilitator must make sure the youth are presented with the
concepts. The facilitator stated he ensures the youth are stating the information correctly and the
youth are required to journal about the presented concepts. The facilitator also stated the youth
on level four are required to run groups. The clinical director stated the facilitator meets with the
youth prior to the scheduled group times to ensure the youth understand the concepts prior to
having them present the material. The technical assistance specialist reaffirmed the groups
should be run by the facilitator and not the youth. The technical assistance specialist also stated
the groups should be ran for at least one hour and showed the clinical director the facilitator
documented the group went from 5p to 6p on the attendance sheet. The group actually ran from
5:06p to 5:32p and should not be entered into EBS as one hour for that lesson.
The program can earn 1 point if if the facilitator adheres to the manual.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 0 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the annual review period of February 1, 2021 through January
31, 2022 shows the program did experience turnover of Impact of Crime facilitators. A case
manager left the program after the cohort ended. The program conducted four cohorts during the
annual review period. The dates of the cohorts were 11/11/2020-3/6/2021, 3/17/2021-6/30/2021,
3/16/2021-7/24/2021, and 7/21/2021-11/15/2021.
The 11/11/2020 cohort began with eight youth and ended with five youth. This cohort began
during the last review period and ran into the current annual review.
The 3/16/2021 cohort began with nine youth and ended with eight youth. The youth who left prior
to completing the cohort was arrested and went to jail. The case manager began the cohort and
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
facilitated alone until 6/10/2021. The recreation therapist then came in and facilitated the group on
6/12/2021 and 6/14/2021. There was a gap in service between 6/14/2021 and 7/21/2021. The
facilitator had to take emergency leave. No other trained facilitator took over during the gap in
service. The cohort ended on 7/24/2021.
The 3/17/2021 cohort began with ten youth and ended with five youth. Five youth were
discharged from the program prior to completing the service. Two attendance sheets showed the
facilitator ran back to back groups on 5/6/2021 and 5/7/2021. The technical assistance specialist
informed the facility administrator and the regional clinical director the service should have at least
one day in between each session.
The 7/21/2021 cohort began with seven youth and ended with five youth. One youth was
discharged on 9/28/2021 and another was discharged on 10/25/2021 prior to completing IOC.
There was a gap in service from 11/15/2021 through 2/5/2022. The clinical director stated the
program faced major staff turnover and did not have enough facilitators to run groups. There was
also an issue with COVID during the first two weeks of January 2022. Most of the case managers
and youth care workers were out due to COVID.
The facility administrator stated the process, in the event of facilitator turnover, is the program
plans to implement a co-facilitator guideline in which there are two facilitators running each group.
In the event one of the facilitators is out for any reason, the group can continue.
The technical assistance specialist informed the current clinical director the primary service must
have a day in-between groups. The clinical director stated the facility administrator told her it was
best practice to have a day in-between, but it was not required. The technical assistance
specialist reaffirmed it is a requirement and not simply best practice.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation reviewed for the annual review period of February 1, 2021 through January
31, 2022 shows the program completed internal fidelity monitoring eleven times for facilitators of
Impact of Crime (IOC). The program is missing a monthly fidelity monitoring for one case
manager for the months of March, May, and June 2021. The program did not provide any internal
fidelity monitorings for the months of November 2021, December 2021, and January 2022. The
clinical director stated the fidelity monitorings were missed because the program was not running
an IOC group due to facilitator turnover. The monthly fidelity monitorings were completed by the
director of case management, transition manager, case manager, and the recreation therapist.
The facility administrator stated the program conducts internal fidelity monitoring monthly.
Facilitators trained must be monitored monthly by another trained and experienced facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The documentation reviewed for the annual review period of February 1, 2021 through January
31, 2022 shows the program did not identify any corrective actions on the monthly internal fidelity
monitorings of Impact of Crime. The facility administrator stated the program’s in-house corrective
action process is when corrective action is needed, the facilitator to meet with the clinical director
to review findings from the completed fidelity monitoring to address concerns and provide support.
The facilitator will be re-assessed at the next month’s fidelity monitoring.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation of the annual review period shows the program did not provide an
updated annual evaluation for the case manager and the transition services manager. The clinical
director stated the case manager’s updated evaluation will be completed within sixty days. The
program provided an annual evaluation for the case manager from the prior review period. The
evaluation has an area titled facilitator skill delivering evidence-based practices. The one
evaluation had remarks that specifically discussed the facilitator’s skill in delivering the primary
service.
The program can earn 1 point if each facilitator’s performance evaluation includes discussion of
their skill in delivering this primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 41 youth in the
sample, 17% (7 of 41) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 24 contact hours for this type of service. Of the 41 youth in the
sample, 34% (14 of 41) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 85%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 18 youth
High = 16 youth
Total Youth in Sample = 41 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 39%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 16 youth
Total Youth in Sample = 41 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 85% (35 of 41) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 39% (16 of 41) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 55 85 65%
This SPEP report evaluates Impact of Crime, an intervention delivered at St. Johns Youth
Academy.
56 youth completed the program during the review period. Of those youth, 41 were included in the
sample.
The program scored Medium for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 41 total youth sampled, only
15 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 7
received at least the recommended weeks of service. Youth in the sample completed between 2
and 30 weeks of service, with an average of 12 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 2 points for Amount of Service: Contact Hours. Of the 41 total youth
sampled, only 31 included dosage in the EBS Module. Of those youth with correct dosage, 14
received at least the recommended hours of service. Youth in the sample completed between 2
and 60 hours of service, with an average of 22 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT). This score reflects youths' most recent
CAT score prior to placement at the program. The program itself has no control over youths' CAT
risk level because the scored assessment was administered prior to the youths' admission.
RECOMMENDATION(S):
St. Johns Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
the facilitator adheres to the manual, there are no unnecessary gaps in service, fidelity monitoring
is completed every month for each facilitator, and each facilitator’s performance evaluation
includes discussion of their skill in delivering this primary service.
St. Johns Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 SJYA IOC.pdf

service_type: Impact of Crime