summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Impact of Crime
SPEP Review Date(s): 1/29/19‐2/1/19
Florida Department of Juvenile Justice
Report Date(s): 3/6/2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime. The program was awarded 15 points
because the Primary Service is identified as a Group 3 Service. The specific Sub-Component
Service Type identified is Social Skills Training. The Primary Service was identified as this type of
service as it focuses on developing social skills required for an individual to interact in a positive
way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Impact of Crime as a primary service for the review year. A review of the
program's documentation from the twelve-month review period revealed seven staff were trained
to conduct Impact of Crime groups. The documentation showed there were four cohorts ran
during the review period. The training is a twenty-eight-hour training conducted over a four-day
period by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Impact of Crime: Addressing the Harm manual, which was provided for
review. The manual opens with an introduction to the Impact of Crime Curriculum. The manual
goes over all seven chapters of the group and provides a script for the facilitator to follow. The
manual also has an appendix and a student guide with all the student exercises.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The latest Impact of Crime group concluded on January 26th. This curriculum was not able to be
observed; therefore, this indicator rates as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was staff turnover for the review period; however, the turnover did not affect the group.
There were four cohorts run during the review period, with no gaps in service noted in the
documentation.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the program showed the program completed fifteen internal
fidelity monitoring during the twelve-month review period. The fidelity monitoring was completed
each month for each facilitator, without exception. An informal interview with the Clinical Director
indicated the trained facilitators take turns completing fidelity checks on each other. The clinical
director stated that he wanted to be trained in Impact of Crime, so he could begin conducting
fidelity checks.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An informal interview with the clinical director confirmed the program has a process for corrective
actions based on fidelity monitoring. Monthly fidelity checks completed on each facilitator. If a
corrective action is noted during the fidelity, the Clinical Director will discuss it with the facilitator
and then re-evaluate the corrective action during the next fidelity monitoring. There was no
corrective action noted during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program only had three staff evaluations available for
review. One of the staff members was not at the program long enough for an evaluation. The
evaluations have an area for evidence-based services skill demonstration. All reviewed
evaluations contained information on the specific evidence-based program skill demonstration.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 62 youth in the
sample, 47% (29 of 62) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 62 youth in the
sample, 53% (33 of 62) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 18 youth
High = 37 youth
Total Youth in Sample = 62 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 60%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 37 youth
Total Youth in Sample = 62 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (58 of 62) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 60% (37 of 62) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime, an intervention delivered at St. John's Youth
Academy.
67 youth completed the program during the review period. Of those youth, 62 had not previously
received this intervention in another setting.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 62 total youth sampled, only
29 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 29
received at least the recommended weeks of service. Youth in the sample completed between 16
and 19 weeks of service, with an average of 18 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 4 points for Amount of Service: Contact Hours. Of the 62 total youth
sampled, only 36 included dosage in the EBS Module. Of those youth with correct dosage, 33
received at least the recommended hours of service. Youth in the sample completed between 7
and 43 hours of service, with an average of 35 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8


state: FL

facility_name: Saint Johns Youth Academy

summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Skillstreaming the Adolescent
SPEP Review Date(s): 1/29/19‐2/1/19
Florida Department of Juvenile Justice
Report Date(s): 3/5/2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Skillstreaming the Adolescent. The program was
awarded 15 points because the Primary Service is identified as a Group 3 Service. The specific
Sub-Component Service Type identified is Social Skills Training. The Primary Service was
identified as this type of service as it focuses on developing social skills required for an individual
to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Skillstreaming the Adolescent as a primary service for the review year. A
review of the program's documentation from the twelve-month review period confirmed nine staff
were conducting groups during the review period. The program has a written training protocol for
the Skillstreaming the Adolescent service. The training is a four-hour in-service training
conducted by the clinical director.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized Skillstreaming the Adolescent, 3rd edition. The manual was provided for
review. The manual gives an introduction to Skillstreaming, contains scripts for the facilitators to
follow as well as specific instructions for each skill taught, example role plays, and student
exercises. There are no separate student manuals. The facilitator makes copies of the lesson
and hands out those copies to the youth in the group. There are 50 skills to be taught to the
youth. Skillstreaming is an open group.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A group facilitation of Skillstreaming was observed on Tuesday at 2:15p. The group was working
on skill sixteen (16), Expressing Your Feelings. The facilitators had skill steps written on the
whiteboard. The facilitators began the group with a rule review. They went into reviewing the
prior skill, Negotiating. The facilitators asked the youth if they had an opportunity to use the
negotiating skill. The facilitators then passed out a handout of emotions. The facilitators
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
demonstrated the skill using role plays and then asked the youth to role play. The group was
assigned a skill step to look for during the role plays and then inform the group when they noticed
the skill. The facilitators ended the group by assigning homework.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was staff turnover for the review period; however, the turnover did not affect the group.
There was one staff member on maternity leave from the end of August 2018 to the end of
November 2018. This did not affect the groups. Another trained facilitator took over facilitating
the group for the therapist on maternity leave. Fifteen (15) cohorts were run during the review
period, with no gaps in services.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the program showed that the program completed 52 internal
fidelity monitorings during the twelve-month review period. The fidelity monitoring was completed
five times in February 2018; seven times in March 2018; six times in June 2018; five times each in
July, August, and September 2018; four times each in October and November 2018; five times in
December 2018; and six times in January 2019. There was no fidelity monitoring completed for
April 2018 and May 2018. The clinical director stated he conducts the fidelity monitoring.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on fidelity monitoring.
The Clinical Director conducts monthly fidelity monitoring. If a corrective action is noted during
the fidelity, the Clinical Director will discuss it with the facilitator and then re-evaluate the
corrective action during the next fidelity monitoring. There was a corrective action noted on the
fidelity monitoring on 6/5/18 which was followed up on the next fidelity monitoring on 7/19/18. The
Clinical Director noted the corrective action was remediated. The fidelity monitoring sheet on
11/1/18 had several poor marks, but no corrective action. The Clinical Director indicated he
marked them "poor" due to the youth acting out during group. The Clinical Director stated he
discussed it with the facilitator and determined no corrective action was needed because it was
out of the facilitator's control.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program only had nine staff evaluations available for
review. The evaluations have an area for evidence-based services skill demonstration. All
reviewed evaluations contained information on the specific evidence-based program skill
demonstration.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 16 weeks for this type of service. Of the 67 youth in the
sample, 81% (54 of 67) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 67 youth in the
sample, 97% (65 of 67) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 20 youth
High = 40 youth
Total Youth in Sample = 67 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 60%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 40 youth
Total Youth in Sample = 67 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (63 of 67) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 60% (40 of 67) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 23 25 92%
Totals 79 85 93%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at St. John's
Youth Academy.
67 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 67 total youth sampled, only
58 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 54
received at least the recommended weeks of service. Youth in the sample completed between 13
and 61 weeks of service, with an average of 32 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 67 total youth
sampled, all 67 included dosage in the EBS Module. Of those youth, 65 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 22 and 103 hours of
service, with an average of 60 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
fidelity monitoring is completed every month for each facilitator.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Saint Johns Youth Academy

program_status_note: Closed

service_type: Skillstreaming the Adolescent


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. Johns Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Thinking for a Change
SPEP Review Date(s): 1/29/19‐2/1/19
Florida Department of Juvenile Justice
Report Date(s): 3/5/2019
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change. The program was awarded 30
points because the Primary Service is identified as a Group 5 Service. The specific Sub-
Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service was
identified as this type of service as it is intended to correct faulty cognitions and perceptions and
provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Thinking for a Change (T4C) as a primary service for the review year. A
review of the program's documentation from the 12-month review period showed six staff had
formal training. The training is a thirty-two hour training completed by a qualified trainer. The
documentation showed that there were four staff conducting groups during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Thinking for a Change 4.0 manual. The manual was provided for review.
The manual includes 25 lessons. Each lesson has a script for the facilitator and instructions for
each activity. The manual requires that the lessons be given in order. The manual provides
specific instructions on the implementation of the program. There are no separate student
workbooks. The facilitators make copies of the current lessons to hand out to the youth. There
was documentation of three cohorts being ran during the review period. There were no sign in
sheets provided for the months of January 2018 through July 2018; however, there were fidelity
monitoring sheets completed for the months of February through May, 2018.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A group facilitation of Thinking for a Change was observed on Tuesday at 6:05. The lesson
completed during the group was Lesson Six, Cognitive Self-Change. The facilitator began group
with a check in and then went over the homework from the last session, Knowing Your Feelings.
The facilitator opened the group with a game to get the group's attention. The facilitator began
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
the current lesson. The facilitator had the manual out and opened to the lesson and was reading
the script provided by the manual. There was a lot of side talk going on during the group. There
were a few youth trying to make jokes as well. The facilitator was able to get the youth back on
track and finish with the lesson.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was staff turnover during the review period. The documentation provided showed one
cohort successfully completed, and there were two groups running at the time of the review.
There were attendance sheets for the cohorts beginning on 8/7/18-12/27/18, 9/18/18-current, and
1/8/19-current. There were no sign in sheets provided that documented groups being ran for
January 2018-July 2018.
The program can earn 2 points if there are no unnecessary gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the program showed that the program completed ten internal
fidelity monitorings over the course of seven months of the twelve-month review period. The
fidelity monitoring was completed on 2/13/18, 3/15/18, 4/17/18, 5/8/18, 10/23/18, 10/25/18,
11/6/18, 11/9/18, 12/18/18, and 12/20/18. The Director of Case Management (DCM) conducts
fidelity monitoring for Thinking for a Change. The DCM was out on medical leave during the
summer of 2018 and was unable to complete the monthly fidelity monitoring during that time.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on fidelity monitoring.
The Director of Case Management conducts monthly fidelity monitoring and will then meet with
the facilitator if there is an identified corrective action. The Director of Case Management will
follow up on the corrective action during the next fidelity monitoring. There were no corrective
actions noted for this review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of documentation showed the program only had three staff evaluations available for
review. The evaluations were completed on the case managers. For two case managers, the
evaluation has an area for evidence-based services; however, there was nothing noted for the
Thinking for a Change group. Another case manger's evaluation did have specific information
pertaining to their facilitation of the Thinking for a Change group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service ‐ Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 64 youth in the
sample, 33% (21 of 64) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 45 contact hours for this type of service. Of the 64 youth in the
sample, 38% (24 of 64) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 94%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 3 youth
Moderate-High = 19 youth
High = 38 youth
Total Youth in Sample = 64 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 59%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 38 youth
Total Youth in Sample = 64 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 94% (60 of 64) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 59% (38 of 64) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 23 25 92%
Totals 82 100 82%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St. John's Youth
Academy.
67 youth completed the program during the review period. Of those youth, 64 had not previously
received this intervention in another setting
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 64 total youth sampled, only
22 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 21
received at least the recommended weeks of service. Youth in the sample completed between 14
and 23 weeks of service, with an average of 21 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 2 points for Amount of Service: Contact Hours. Of the 64 total youth
sampled, only 25 included dosage in the EBS Module. Of those youth with correct dosage, 24
received at least the recommended hours of service. Youth in the sample completed between 39
and 58 hours of service, with an average of 52 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
there are no unnecessary gaps in service, and by ensuring fidelity monitoring is completed every
month for each facilitator.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Thinking for a Change


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Impact of Crime
SPEP Review Date(s): January 7-10, 2020
Florida Department of Juvenile Justice
Report Date(s): April 15, 2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Impact of Crime as a primary service for the review year. A review of the
program's documentation from the twelve-month review period showed three staff were trained to
conduct Impact of Crime groups. The documentation showed there were four cohorts completed
during the review period. There are two cohorts currently in progress. The training is a
twentyeight-hour training conducted over a four-day period by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime: Addressing the Harm v 2.0 as the manual. The
facilitator’s manual is divided into seven chapters, an appendix with handouts, and the student
guide section (which includes the student exercises). The manual addresses the order in which to
deliver the content and gives the facilitators an outline to follow. The facility does not have
separate student workbooks to give out. The facilitators make copies of the exercises before each
session to pass out to the youth. The group is a closed group with no more than twelve youth in
each cohort. The curriculum is to be delivered to the youth over a minimum of sixteen weeks for
thirty-two sessions.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime group was observed on 1/15/2020. Group began at 3:50p and ended at
4:43p. The facilitator began group by passing out the youth’s folders. The youth had
approximately eight minutes to study for the group’s review/quiz. After the youth reviewed
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
chapters one through three on their own, the facilitator did a group review. The facilitator placed
the four victim impacts and the four stinking thinking errors on the walls around the room. The
facilitator then read a scenario to the youth and asked them to go to the impact area or stinking
thinking that best described that scenario. Each youth was asked to justify their reasons for
choosing the area. For instance, the youth stood under the word that said minimizing, the youth
would have to justify why the scenario was minimizing. After the review, the facilitator passed out
the quiz to the youth and gave them time to complete it. Once the youth completed the quiz, the
facilitator went over the answers with the youth. The facilitator asked the youth what reparation
project they would like to complete for this cohort. The youth came up with three ideas. The
facilitator said that she would begin the process to gain permission and see which one of the three
ideas the group would be able to complete.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There was staff turnover for the review period; however, the turnover did not affect the
delivery of Impact of Crime. There were six cohorts run during the review period. There were no
gaps in service noted in the documentation.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation for the program showed the program completed fourteen internal
fidelity monitorings during the twelve-month review period. The fidelity monitoring was completed
at least monthly for each facilitator, without exception. The clinical director stated the trained
facilitators take turns completing fidelity checks on each other.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on fidelity monitoring.
There are monthly fidelity checks completed on each facilitator. If a corrective action is noted
during the fidelity, the Clinical Director will discuss it with the facilitator and then re-evaluate the
corrective action during the next fidelity monitoring. There was no corrective action noted during
the review period.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed the program had three staff facilitate groups and each staff
had evaluations available for review. The evaluations have an area for evidence-based services
skill demonstration. All reviewed evaluations contained information on the specific evidence-
based program skill demonstration.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 63 youth in the
sample, 19% (12 of 63) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 24 contact hours for this type of service. Of the 63 youth in the
sample, 41% (26 of 63) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 23 youth
High = 34 youth
Total Youth in Sample = 63 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 54%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 34 youth
Total Youth in Sample = 63 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (58 of 63) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 54% (34 of 63) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 23 25 92%
Totals 67 85 79%
This SPEP report evaluates Impact of Crime, an intervention delivered at St. John's Youth
Academy.
65 youth completed the program during the review period. Of those youth, 63 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 0 points for Amount of Service: Duration. Of the 63 total youth sampled, only
25 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 12
received at least the recommended weeks of service. Youth in the sample completed between 9
and 23 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 63 total youth
sampled, only 32 included dosage in the EBS Module. Of those youth with correct dosage, 26
received at least the recommended hours of service. Youth in the sample completed between 23
and 43 hours of service, with an average of 30 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Impact of Crime


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: SkillStreaming the Adolescent
SPEP Review Date(s): January 7-10, 2020
Florida Department of Juvenile Justice
Report Date(s): April 15, 2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is SkillStreaming the Adolescent.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Skillstreaming the Adolescent as a primary service for the review year. A
review of the program's documentation showed twelve staff were conducting groups during the
review period. All twelve staff were trained in the primary service prior to delivering services to
youth. The program has a written training protocol for the Skillstreaming the Adolescent service.
The training is a four-hour in-service training conducted by the clinical director. The training is
broken down into three parts. The first part is the introduction to Skillstreaming the Adolescent.
This section is one hour in length. The second section is the review of materials. This section is
two hours in length. The third section is the observation by the clinical director of the trained staff
conducting a group session.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Skillstreaming the Adolescent, 3rd edition. The manual gives an
introduction to Skillstreaming the Adolescent, contains scripts for the facilitators to follow as well
as specific instructions for each skill taught, example role plays, and student exercises. There are
no separate student manuals. The facilitator makes copies of the lesson and hands out those
copies to the youth in the group. There are fifty skills taught to the youth. Skillstreaming the
Adolescent is an open group, with groups being ran year-round.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Skillstreaming the Adolescent was observed. The group began its session at 3:30p
and ended at 4:26p; the technical assistance specialist joined the group shortly after the group
began. The facilitator began group by asking each youth to take turns stating one norm. The
facilitator then asked the youth to take turns stating what they learned during the last session. The
facilitator moved into the current lesson of “Knowing Your Feelings.” The facilitator asked the
youth how they know their feelings. The facilitator then described the skill steps he had written on
the white board. He then had the youth state each step out loud together as a group. The first two
youth did not complete the role play correctly. The facilitator stopped the youth and had the youth
correct the mistake. The incorrect youth was using the thought bubble and the facilitator made the
corrections and allowed the youth to continue. The facilitator let the youth pick the first two role
play topics he then chose the last two for the youth. The facilitator asked the youth what they felt
they learned when they were presenting the skill.
One youth walked out of the group and went into the next room with other staff and youth. The
facilitator did not address the youth who left the group. The clinical director was conducting a
fidelity monitoring of the group. After the group ended, the technical assistance specialist asked
the clinical director if he thought the facilitator knew the youth left the group. The clinical director
stated he would address that question with the facilitator. The clinical director and the facilitator
then informed technical assistance specialist the facilitator was aware the youth left the group, the
youth was having a rough day and the facilitator was not wanting to escalate any behavioral
issues.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation for the twelve-month review period showed there was staff
turnover for the review period which did affect the groups, but there were no gaps in services.
There are three staff facilitating groups who are no longer employed with the program; the youth
who were assigned to those staff were moved into another group with another
facilitator. The youth were assigned several different facilitators during the review period.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of the documentation for the program for the twelve-month review period showed the
program completed sixty-five internal fidelity. The fidelity monitoring was completed at least
monthly for each facilitator, without exception.
The clinical director completes the internal fidelity monitoring monthly. The clinical director reviews
the fidelity monitoring with the facilitator and then implements any corrective action as needed.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on fidelity monitoring.
The clinical director will conduct monthly fidelity monitoring. If a corrective action is noted during
the fidelity, he will then discuss it with the facilitator and then re-evaluate the corrective action
during the next fidelity monitoring.
The documentation reviewed for the twelve-month review period shows the clinical director noted
corrective actions were needed on fourteen of the sixty-five internal fidelity monitorings. The
clinical director stated all of the corrective actions for the review period have been addressed and
ameliorated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed the program only had five staff evaluations available for
review. There was a total of twelve staff who conducted groups during the review period. Seven of
the staff are no longer employed at the program. The evaluations have an area for evidencebased
services skill demonstration. All reviewed evaluations contained information on the specific
evidence-based program skill demonstration.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 64 youth in the
sample, 48% (31 of 64) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates a target of 24 contact hours for this type of service. Of the 64 youth in the
sample, 78% (50 of 64) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 23 youth
High = 34 youth
Total Youth in Sample = 64 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 55%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 35 youth
Total Youth in Sample = 64 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (60 of 64) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 63% (35 of 64) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 23 25 92%
Totals 73 85 86%
This SPEP report evaluates Skillstreaming the Adolescent, an intervention delivered at St. John's
Youth Academy.
65 youth completed the program during the review period. Of those youth, 64 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 4 points for Amount of Service: Duration. Of the 64 total youth sampled, only
41 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 31
received at least the recommended weeks of service. Youth in the sample completed between 1
and 77 weeks of service, with an average of 26 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 6 points for Amount of Service: Contact Hours. Of the 64 total youth
sampled, only 59 included dosage in the EBS Module. Of those youth with correct dosage, 50
received at least the recommended hours of service. Youth in the sample completed between 2
and 84 hours of service, with an average of 36 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


summary: SPEP - FY19-20

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
St. John's Youth Academy
Sequel Youth and Family Services
(Contract Provider)
4500 Avenue D
St. Augustine, Florida 32085
Primary Service: Thinking for a Change
SPEP Review Date(s): January 7-10, 2020
Florida Department of Juvenile Justice
Report Date(s): April 15, 2020
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Thinking for a Change.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilized Thinking for a Change as a primary service for the review year. A review of
the program's documentation from the twelve-month review period showed six staff had formal
training. The training is a thirty-two-hour training completed by a qualified trainer. The
documentation showed there were three staff conducting groups during the review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilized the Thinking for a Change 4.0 manual. The manual includes twenty-five
lessons. Each lesson has a script for the facilitator and instructions for each activity. The manual
requires the lessons be given in order. The manual provides specific instructions on the
implementation of the program. There are no separate student workbooks. The facilitators make
copies of the current lessons to hand out to the youth.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Thinking for a Change was observed during the annual SPEP review. There were
three youth in the group. The facilitator stated there were four youth discharged from the program
who were initially in the cohort. The facilitator asked the youth how they know what their feelings
are and went over the three steps of knowing your feelings. The facilitator then asked the youth if
each step was a thinking step or an action step. The facilitator went into the role plays. One youth
faced the wall away from the group while they were doing the role plays, stating he didn’t want to
participate in the group. The facilitator passed out homework at the end of the session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The documentation reviewed for the twelve-month review period showed there was staff turnover,
but the turnover did not affect facilitation of the groups. There were four cohorts run during the
review period. The facilitators remained the same in each of the cohorts. The program has a
process in place to avoid gaps in service due to facilitator turnover.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
A review of documentation showed the program completed thirteen (13) internal fidelity
monitorings during the twelve-month review period. Fidelity monitoring was completed each
month for each facilitator. The internal fidelity monitoring was completed by staff trained in
Thinking for a Change.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a process in place to address corrective actions based on fidelity monitoring. A
trained facilitator will conduct monthly fidelity monitoring. The director of case management or the
clinical director will then meet with the facilitator if there is an identified corrective action. The
director of case management or the clinical director will follow up on the corrective action during
the next fidelity monitoring. There were no corrective actions noted for this review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include evaluation
of skill in delivering the intervention/service.
A review of documentation showed the program had three staff evaluations available for review.
The evaluations were completed on the three staff members who facilitated Thinking for a
Change. There is an area on the evaluation which addresses the facilitators skill in delivering the
primary service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service - Duration
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates the target duration of 15 weeks for this type of service. Of the 62 youth in the
sample, 35% (22 of 62) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 45 contact hours for this type of service. Of the 62 youth in the
sample, 10% (6 of 62) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 23 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 92%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 92%
Moderate to High Score: 10 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 83%
Table 1
Moderate = 1 youth
Moderate-High = 24 youth
High = 32 youth
Total Youth in Sample = 62 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 52%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 32 youth
Total Youth in Sample = 62 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 92% (57 of 62) youth scored Moderate to High-Risk to reoffend, for a score
of 10 points.
Of the SPEP sample, 52% (32 of 62) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 35%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 2 10 20%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 23 25 92%
Totals 80 100 80%
This SPEP report evaluates Thinking for a Change, an intervention delivered at St. John's Youth
Academy.
65 youth completed the program during the review period. Of those youth, 62 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 2 points for Amount of Service: Duration. Of the 62 total youth sampled, only
29 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 22
received at least the recommended weeks of service. Youth in the sample completed between 3
and 31 weeks of service, with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 0 points for Amount of Service: Contact Hours. Of the 62 total youth
sampled, only 37 included dosage in the EBS Module. Of those youth with correct dosage, 6
received at least the recommended hours of service. Youth in the sample completed between 10
and 53 hours of service, with an average of 33 hours.
The program was awarded 23 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
St. John's Youth Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
St. John's Youth Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8