summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Seven Challenges
SPEP Review Period: December 15, 2019 - December 14, 2020
SPEP Review Conducted: December 15, 2020 - December 17, 2020
Florida Department of Juvenile Justice
Report Date: March 9, 2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program trains therapuetic staff to facilitate the Seven Challenges curriculum. All therapy staff
are trained to conduct the curriculum. Due to layoffs at the facility during the last quarter of the
review period, there are three trained staff at this time. One therapist is the main facilitator for the
Seven Challenges curriculum.
All training records were provided for this review. All staff have been formally trained to conduct
Seven Challenges for the youth in the program.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided three manuals for The Seven Challenges curriculum. The facilitator's
manual has thirteen chapters, which covers implementation, each of the seven challenges,
counseling approaches, gender issues, supervision, games for groups, two appendices, four extra
worksheets, group note examples, and treatment goal examples. There is a section in the manual
for group session topics. The topics are able to be copied. The youth read through the topic and
they discuss how the topic applies to their life. The curriculum is open for the therapist to be
creative in how they want to facilitate the group. The Seven Challenges is designed to be an open
discussion for all youth to participate and apply the lesson to their lives.
The second manual provided for review was an activity book. The activity book provides various
counseling activities for each challenge. Each activity provides materials, instruction, script, how
to connect the lesson, and worksheets for each lesson.
The last manual was the journaling book the youth use during journaling sessions. One session
per week is typically a journaling session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Seven Challenges is held as an open group and it is run consistently at the program. As an open
group the youth complete the curriculum at their own pace. Youth are able to begin this group
when they enter the program. The curriculum is provided to all youth with a substance abuse
diagnosis and are placed in the substance abuse part of the program.
Groups are held Tuesdays and Thursdays from 10:30am to 11:30 am and 11:30am to 12:30pm.
All documentation for the group is maintained in a binder located in the clinical director's office
and in the Evidenced Based Services module in the Juvenile Justice Information System.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Seven Challenges group was observed during second quarterly review (March to June). The
group was observed on Tuesday, June 9, 2020 from 10:30am to 11:30am. There were eight youth
in the group, one facilitator, and two direct care staff.
The facilitator maintained an upbeat and positive group setting. She also utilized motivational
interivewing skills to encourage discussion from the youth. This reviewer was provided a copy of
the lesson. The group was on challenge six: lapse and relapse. The group worked through a
relapse prevention plan after the reading and discussion. The youth were supportive of each other
and encouraged each other throughout the group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
In the event of a facilitator being out or leaving, another trained faciltiator continues with the
group. The program experienced facilitator turnover during the last quarter of the year (October to
December) due to layoffs at the program. There is currently one substance abuse therapist at the
time of this report. Should another therapist be out or leave, the clinical director or other trained
therapist can continue with the group. The turnover did not cause any gaps in service.
All sign-in sheets were provided for the review period. During the month of October two sessions
were held due to two youth being in quarantine. Seven Challenges groups were unable to be held
on the dorms during this time so exposure to other youth and staff were limited. This did not
create a large gap in service. Once the youth were medically cleared, groups were able to
continue as normal. The program has experienced a low census this year and the amount of
youth in group is low; however, this has been approved in order for treatment groups to be able to
continue.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Internal fidelity monitoring for Seven Challenges is conducted by the clinical director for the
program. All internal fidelity monitoring forms were provided for the twelve month review period.
There were no missing internal fidelity monitoring forms during the twelve month review period.
The internal fidelity monitoring forms include specific details regarding facilitators, corrective
actions, and feedback by the facilitator and observer for the group. The forms include names,
dates, and times of the group.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director observes the Seven Challenges groups and addresses any issues or
corrective actions. These are documented on the internal fidelity monitoring forms along with any
follow-up information. If there is an issue or corrective action it is documented on the internal
fidelity monitoring form. The clinical director will conduct another observation and document it on
the internal fidelty monitoring form. If the issue continues further action is taken and documented
on the internal fidelity monitoring form.
There were no corrective actions noted on the internal fidelity monitoring forms reviewed.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual evaluation forms include a section titled "Therapeutic Skills" and includes specific
programming in this section for the facilitator. The facilitator for Seven Challenges received the
annual performance evaluation on March 9, 2020 with no issue.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 23 youth in the
sample, 65% (15 of 23) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 2 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 20%
Research indicates a target of 40 contact hours for this type of service. Of the 23 youth in the
sample, 22% (5 of 23) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 70%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 4 youth
Moderate-High = 10 youth
High = 2 youth
Total Youth in Sample = 23 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 9%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 2 youth
Total Youth in Sample = 23 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 70% (16 of 23) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 9% (2 of 23) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 2 10 20%
Risk Level of Youth Served 5 25 20%
Totals 63 95 66%
This SPEP report evaluates Seven Challenges, an intervention delivered at DOVE Academy.
43 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 23 total youth sampled, only
2 included dosage with end dates in the EBS Module, as a result of an administrative oversight. A
manual audit of the remaining youth was conducted to include dosage calculations. Of those
youth, 15 received at least the recommended weeks of service. Youth in the sample completed
between 18 and 45 weeks of service, with an average of 29 weeks.
The program earned 2 points for Amount of Service: Contact Hours. Of the 23 total youth
sampled, only 3 included dosage in the EBS Module, as a result of an administrative oversight. A
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
manual audit of the remaining youth was conducted to include dosage calculations. Of those
youth, 5 received at least the recommended hours of service. Youth in the sample completed
between 14 and 55 hours of service, with an average of 32 hours.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 87%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Dove Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Dove Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9


state: FL

facility_name: DOVE Academy

summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Impact of Crime
SPEP Review Period: December 15, 2019 to December 14, 2020
SPEP Review Conducted: December 15, 2020 to December 17, 2020
Florida Department of Juvenile Justice
Report Date: March 9, 2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Case management staff is formally trained to facilitate the Impact of Crime (IOC) curriculum.
There are currently six trained staff: three case managers, one shift supervisor, the transition
specialist, and the program director.
Four of the staff members have faciliated IOC groups during the twelve month review period. All
training records have been provided for review and all were formally trained.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the current Impact of Crime (IOC) facilitator's manual. There are four
sections to the IOC manual. The first section is the facilitator’s guide, which includes an
introduction and implementation section. This also includes how the chapters and lessons are
structured and how they should be delivered. There are two parts of the first section that goes in
depth and discusses the importance of victims and considerations when facilitating IOC to girls.
The second section has all seven chapters divided and numbered. Within each chapter contains
the IOC script, lessons, and activities for the youth. Every chapter begins with objectives, words to
know, and definitions. The third section is the appendix. This contains extra worksheets, list for
videos, behavior contracts, and other helpful worksheets to use during group. The final section is
the student guide. This is the specific workbook for the youth. These can be copied and provided
to the youth.
Impact of Crime is held on Tuesdays and Thursdays from 3:15pm to 4:15pm. The facilitators
rotate conducting the groups throughout the review period. The documents for group are
maintained in the program director's office. All youth are eligible to receive the IOC curriculum.
Impact of Crime is held as a closed group, meaning when a youth enters the program they will
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
need to wait for a new cohort to start. No youth are able to be added to a group once lesson one
has been completed.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
An Impact of Crime (IOC) group was observed virtually on Tuesday, September 1, 2020 from
3:15pm to 4:15pm. There were six youth in attendance for this group. The lesson was part of
chapter 5; looking at the ripple effect of crime. The youth were interactive and participated in all
activities and discussions. The facilitator utilized the white board, youth workbooks, and had an
article for the youth to review in class. The group discussed some heavy topics during this
session. A therapist was also made available for the youth if they felt they needed to speak with
someone. There was no co-facilitator for this group. There were excellent interactions and
discussion on gangs. Youth took turns reading and had a mature discussion on the topics
covered. The faciltiator did a great job breaking down the information and circled back to prior
chapters to review. The facilitator encourages youth and utilizes Motivational Interviewing
techniques throughout group.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program's policy is for another trained facilitator to continue facilitating a group should a
facilitator be out or leave the program. The program has not experienced turnover with Impact of
Crime facilitators during this review period. All sign-in sheets for the cohorts conducted were
provided for this review.
There were five cohorts completed during the twelve month review period. Many of the groups
had less than the required number due to a low census; however, this was approved by the
Department due to Covid-19 protocols.
The first cohort was held from July 31, 2019 to December 12, 2019. This cohort had eight youth to
start and seven to complete. One youth was released early and did not complete the group. The
second cohort was held from September 23, 2019 to February 3, 2020. There were eleven youth
to begin and eight completed. One youth was released prior to completion due to medical issues
during the cohort. Two youth were released early from the program and unable to successfully
complete the curriculum.
The third cohort was held December 5, 2019 to April 13, 2020. The group started with eleven
youth and six youth completed. Two were released due to medical issues. There other three were
all released early. The fourth cohort was held from February 11, 2020 to June 17, 2020. The
group started with nine youth and ended with six youth. Three youth were released early due to
release dates being moved up by the committing courts. The facilitators conducted 1:1 sessions
and provided attendance sheets for those sessions. The three youth were able to complete
Impact of Crime prior to their early releases.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The fifth cohort was started on June 25, 2020 and ended on Spetember 22, 2020. The group
started with seven youth and six youth completed the group. One youth was released early and
did not successfully complete the curriculum.
A sixth cohort was started on September 18, 2020 with eight youth and is continuing to meet at
this time. This group did complete the new pre-test for the Impact of Crime curriculum. There are
six youth in this group currently. Two other youth started in this group and were removed from the
group due to disruptive behaviors. These youth had to wait for the next cohort which started
December 1, 2020 and were able to participate in that cohort.
A seventh cohort was started on December 1, 2020 and is continuing to meet at this time. The
group has ten youth in it and they completed the pre-test for this curriculum.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program director or transition specialist conducts internal fidelity monitoring for Impact of
Crime groups. Three trained staff facilitated the Impact of Crime groups during this twelve month
review period. The program director reviews and signs all internal fidelity monitoring forms.
There were no missing internal fidelity monitoring reports for the twelve month review period.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program's policy is for the program director to address any issues or corrective actions
regarding group facilitation.
There were no corrective actions during this review period. The internal fidelity monitoring forms
are detailed and provided feedback by the observer and allowed the facilitators to provide their
own feedback. There were suggestions made on the internal fidelity monitoring forms on a
consistent basis.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
All four of the Impact of Crime facilitators have received their annual performance evaluations at
this time. The annual performance evaluation includes a section for group facilitation and all four
discuss Impact of Crime facilitation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 43 youth in the
sample, 65% (28 of 43) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 43 youth in the
sample, 84% (36 of 43) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 63%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 6 youth
Moderate-High = 17 youth
High = 4 youth
Total Youth in Sample = 43 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 9%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 4 youth
Total Youth in Sample = 43 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 63% (27 of 43) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 9% (4 of 43) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 5 25 20%
Totals 59 85 69%
This SPEP report evaluates Impact of Crime, an intervention delivered at DOVE Academy.
43 youth completed the program during the review period. Of those youth, 43 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 43 total youth sampled, all
43 included dosage with end dates in the EBS Module. Of those youth, 28 received at least the
recommended weeks of service. Youth in the sample completed between 6 and 20 weeks of
service, with an average of 17 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 43 total youth
sampled, all 43 included dosage in the EBS Module. Of those youth, 36 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 11 and 32 hours of
service, with an average of 29 hours.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Dove Vocational Academy can maintain their SPEP Quality of Service Delivery score by
continuing the practices in place at the time of this review.
Dove Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

source: DJJ

report_type: SPEP

program_name_raw: Dove Academy

program_status_note: Open

service_type: Impact of Crime


summary: SPEP - FY20-21

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Moral Reconation Therapy
SPEP Review Period: December 15, 2019 - December 14, 2020
SPEP Review Conducted: December 15, 2020 - December 17, 2020
Florida Department of Juvenile Justice
Report Date: March 9, 2021
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Moral Reconation Therapy.
The program was awarded 30 points because the Primary Service is identified as a Group 5
Service. The specific Sub-Component Service Type identified is Cognitive Behavioral Therapy.
The Primary Service was identified as this type of service as it is intended to correct faulty
cognitions and perceptions and provides skills individuals can use to monitor thought patterns and
correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
Therapy staff are trained to conduct Moral Reconation Therapy. There have been four formally
trained staff throughout the twelve month review period; no new staff were trained. All three
therapists conducted Moral Reconation Therapy throughout the review period.
All training records were provided and reviewed. The therapy staff were formally trained to
facilitate Moral Reconation Therapy.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided a workbook titled, "Juvenile MRT: How to Escape Your Prison, A Moral
Reconation Therapy Workbook". The book is written by Dr. Gregory Little and Dr. Kenneth
Robinson. For this curriculum, each youth is provided a workbook that they will work in throughout
their time in group. There are twelve steps each youth must work through and successfully
complete in order to successfully complete the curriculum. Each youth must complete all twelve
steps of the curriculum. They are given specific tasks they must complete before they can move
to the next step.
Moral Reconation Therapy is held on Mondays and Wednesdays from 10:30am to 11:30am and
11:30 am to 12:30pm. Documents for the groups are maintained in a binder located in the clinical
director's office and in the Evidenced Based Services module in the Juvenile Justice Information
System. Youth who have a mental health diagnosis and/or a substance abuse diagnosis are
eligible to attend Moral Reconation Therapy groups. The results from the Residential Assessment
for Youth (RAY) are also taken into consideration for this group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A virtual group observation was conducted on Tuesday, September 1, 2020 from 10:15am to
11:15am. Seven youth attend this group. Youth took turns reading. The facilitator was organized
and well prepared for the group. Youth work on packets individually as part of the current lesson.
Positive feedback was provided and a mature discussion was held with the group for the topic.
Moral Reconation Therapy is typically held on Mondays and Wednesdays, but was held on
Tuesday this week for program scheduling purposes.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
In the event of turnover or a therapist being out, another therapist will continue with the group
and/or the clinical director will assist in groups. During the last quarter of the review period
(October to December) the program experienced layoffs. There are curently two therapists
conducting Moral Reconation Therapy at this time. The group run by the laidoff therapist
transitioned to another therapist with no gaps in service.
Moral Reconation Therapy is held as an open group meaning the youth can begin the group upon
intake into the program.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The clinical director conducts all internal fidelity monitoring for Moral Reconation Therapy. The
internal fidelity monitoring forms include specific details regarding facilitators, corrective actions,
and feedback by the facilitator and observer for the group. The forms include names, dates, and
times of the group.
All internal fidelity monitoring forms were provided for the twelve month review period. There were
no missing forms for the entire year.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director observes the Moral Reconation Therapy groups and addresses any issues or
corrective actions. These are documented on the internal fidelity monitoring forms along with any
follow-up information. If there is an issue or corrective action, it is documented on the internal
fidelity monitoring form. The clinical director will conduct another observation and document it on
the internal fidelty monitoring form. If the issue continues further action is taken and documented
on the internal fidelity monitoring form.
There were no corrective actions noted nor taken during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The annual evaluation forms include a section titled "Therapeutic Skills" and includes specific
programming in this section for the facilitator. One of the two yearly performance evaluations were
completed at the time of this review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 15 weeks for this type of service. Of the 43 youth in the
sample, 98% (42 of 43) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 45 contact hours for this type of service. Of the 43 youth in the
sample, 47% (20 of 43) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 5 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 20%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 63%
Moderate to High Score: 5 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 42%
Table 1
Moderate = 6 youth
Moderate-High = 17 youth
High = 4 youth
Total Youth in Sample = 43 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 9%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 4 youth
Total Youth in Sample = 43 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 63% (27 of 43) youth scored Moderate to High-Risk to reoffend, for a score
of 5 points.
Of the SPEP sample, 9% (4 of 43) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 5 25 20%
Totals 72 100 72%
This SPEP report evaluates Moral Reconation Therapy, an intervention delivered at Dove
Vocational Academy.
43 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 43 total youth sampled,
none included dosage with end dates in the EBS Module, as a result of an administrative
oversight. A manual audit of the remaining youth was conducted to include dosage calculations.
Of those youth, 42 received at least the recommended weeks of service. Youth in the sample
completed between 1 and 53 weeks of service, with an average of 30 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
The program earned 4 points for Amount of Service: Contact Hours. Of the 43 total youth
sampled, only 17 included dosage in the EBS Module, as a result of an administrative oversight. A
manual audit of the remaining youth was conducted to include dosage calculations. Of those
youth, 20 received at least the recommended hours of service. Youth in the sample completed
between 27 and 72 hours of service, with an average of 46 hours.
The program was awarded 5 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 92%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
Dove Vocational Academy can optimize their SPEP Quality of Service Delivery score by ensuring
facilitator turnover does not impact groups.
Dove Vocational Academy can optimize their SPEP Amount of Service score by ensuring that
dosage for all youth is recorded accurately in EBS and by ensuring that youth receive the full
targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

service_type: Moral Reconation Therapy


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Seven Challenges
SPEP Review Period: December 15, 2020 - December 14, 2021
SPEP Review Conducted: January 10, 2022 - January 14, 2022
Florida Department of Juvenile Justice
Report Date: 2/23/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges.
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes Seven Challenges as a primary service. There were four staff formally
trained to conduct the Seven Challenges groups; one clinical director and three therapists.
All trained therapy staff have facilitated Seven Challenges groups throughout the year based upon
facilitation needs; however, there were three main staff who conducted Seven Challenges groups.
The clinical director conducts internal fidelity monitoring.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program provided three manuals for The Seven Challenges curriculum. The facilitator's
manual has thirteen chapters, which cover implementation, each of the seven challenges,
counseling approaches, gender issues, supervision, games for groups, two appendices, four extra
worksheets, group note examples, and treatment goal examples. There is a section in the manual
for group session topics. The topics are able to be copied. The youth read through the topic and
discuss how it applies to their life. The curriculum is open for the therapist to be creative in how
they want to facilitate the group. The Seven Challenges is designed to be an open discussion for
all youth to participate and apply the lesson to their lives.
The second manual provided for review was an activity book. The activity book provides various
counseling activities for each challenge. Each activity provides materials, instruction, script, how
to connect the lesson, and worksheets for each lesson.
The last manual was the journaling book the youth use during journaling sessions. One session
per week is typically a journaling session.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of The Seven Challenges group was observed on Tuesday, June 22, 2021 from
10:30am to 11:30am. Ten youth were present for the group, which is held in one of the dorms.
There were two youth care workers, the facilitator, and the clinical director present. The reviewer
was provided a copy of the lesson titled: "Challenge Two: Changing our Feelings and Forgetting
Problems". The youth completed the reading and then completed a two-page worksheet. The
facilitator gave ample time for all activities. The facilitator was engaging with all youth and was
comfortable teaching the lesson. She was attentive to each youth when they were providing their
answers, asking open ended questions, and reflecting and summarizing the youth's responses.
All youth actively participated in the group discussion.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program experienced turnover during this annual review period. One therapist left
employment on June 30, 2021. Another therapist is working on a part-time basis. When turnover
occurs, another trained facilitator continues conducting the groups. The youth are familiar with all
facilitators of the Seven Challenges curriculum.
Seven Challenges is held as an open group and it is run consistently at the program. As an open
group, the youth complete the curriculum at their own pace. Youth are able to begin this group
when they enter the program. The curriculum is provided to all youth with a substance abuse
diagnosis and are placed in the substance abuse part of the program.
Groups are held Tuesdays and Thursdays from 10:30am to 11:30am and 11:30am to 12:30pm.
All documentation for the group is maintained in a binder located in the clinical director's office
and in the Evidence-Based Services module in the Juvenile Justice Information System. A review
of sign-in sheets for the twelve month annual review period confirmed groups were held on
Mondays and Wednesdays for September, but the rest of time the groups were held according to
the program schedule of Tuesdays and Thursdays. Each group was at least one hour in length
and had the required number of youth.
There were two sign-in sheets for the month of January due to the youth and therapist being in
quarantine. The youth were in quarantine from January 2, 2021 to January 19, 2021. One
therapist was out from January 5, 2021 to January 18, 2021 and groups were not held during this
time. The program reports youth and staff were in quarantine due to COVID-19 and groups could
not be held as scheduled. This does not qualify as an “unnecessary gap in service”. This group
continued once the facilitator and youth were medically released.
The program can earn 2 points if facilitator turnover does not impact groups.
At the time of the review, the program did not meet criteria to earn a score of 2.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program has a process to conduct monthly internal fidelity monitoring. The clinical director
conducts internal fidelity monitoring for the Seven Challenges facilitators. All internal fidelity
monitoring forms were provided for the twelve month review period.
Fourteen out of sixteen internal fidelity monitoring forms were completed for facilitators during this
annual review. One fidelity observation was not completed due to the facilitators and youth being
in quarantine during the third quarter of the review year. There was one internal fidelity monitoring
missing for a facilitator in the month of September.
The program can earn 2 points if fidelity monitoring is completed every month for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The clinical director addresses any issues or corrective actions with the facilitators of the Seven
Challenges curriculum. The clinical director also meets with the therapy staff weekly to conduct
weekly supervisons. This documentation was provided for review.
There is a section on the internal fidelity monitoring forms to document any issues or corrective
actions with the facilitators. A review was conducted of all internal fidelity monitoring forms for the
twelve month annual review period. There were no significant issues or corrective actions noted.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program's annual performance evaluations include a section to address facilitation of groups
in the program. There is one facilitator for the Seven Challenges groups who was eligible to
receive a six-month evaluation.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
3. Amount of Service - Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 85% (17 of 20) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 40% (8 of 20) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 7 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 28%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 75%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 4 youth
Moderate-High = 10 youth
High = 1 youth
Total Youth in Sample = 20 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
Percentage of Youth with High-Risk Level to Reoffend: 5%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 1 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 75% (15 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 5% (1 of 20) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 4 10 40%
Risk Level of Youth Served 7 25 28%
Totals 69 95 73%
This SPEP report evaluates Seven Challenges, an intervention delivered at DOVE Academy.
44 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The program earned 8 points for Amount of Service: Duration. Of the 20 total youth sampled, all
20 included dosage with end dates in the EBS Module. Of those youth, 17 received at least the
recommended weeks of service. Youth in the sample completed between 15 and 38 weeks of
service, with an average of 27 weeks.
The program earned 4 points for Amount of Service: Contact Hours. Of the 20 total youth
sampled, all 20 included dosage in the EBS Module. Of those youth, 8 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
recommended hours of service. Youth in the sample completed between 11 and 69 hours of
service, with an average of 34 hours.
The program was awarded 7 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 92%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
DOVE Academy can optimize their SPEP Quality of Service Delivery score by ensuring facilitator
turnover does not impact groups and fidelity monitoring is completed every month for each
facilitator.
DOVE Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9

file_name: FY2021-22 DOVE - 7C.pdf

service_type: Seven Challenges


summary: SPEP - FY21-22

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
DOVE Academy
Twin Oaks Juvenile Development, Inc.
(Contract Provider)
5270 Ezell Road
Graceville, Florida 32440
Primary Service: Impact of Crime
SPEP Review Period: December 15, 2020 - December 14, 2021
SPEP Review Conducted: January 10, 2022 - January 14, 2022
Florida Department of Juvenile Justice
Report Date: 2/23/22
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
• Service Type: service type points are awarded based on the group category that the
service falls within
• Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
• Amount of Service: or “dosage” refers to the amount and duration of services that were
received
• Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime.
The program was awarded 15 points because the Primary Service is identified as a Group 3
Service. The specific Sub-Component Service Type identified is Social Skills Training. The
Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated during quarterly and annual reviews.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 9 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program utilizes the Impact of Crime (IOC) curriculum as a primary service. The case
management staff are trained to facilitate IOC groups and conduct internal fidelity monitoring.
The program had six trained staff during the twelve month review period; the program director,
three case managers, transitional specialist, and a youth care worker supervisor. The youth care
worker supervisor did change positions to a therapist during the twelve month review period and
will be facilitating IOC groups as needed. There were three facilitators during this annual review
period. The transition specialist typically conducts internal fidelity monitoring. All training records
have been reviewed for the staff and all staff were formally trained in the IOC curriculum.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program maintains the current Impact of Crime (IOC) facilitator's manual. There are four
sections to the IOC manual. The first section is the facilitator’s guide, which includes an
introduction and implementation section. This also includes how the chapters and lessons are
structured and how they should be delivered. There are two parts of the first section which go in
depth and discuss the importance of victims and considerations when facilitating IOC to girls.
The second section has all seven chapters divided and numbered. Within each chapter contains
the IOC script, lessons, and activities for the youth. Every chapter begins with objectives, words to
know, and definitions. The third section is the appendix. This contains extra worksheets, list for
videos, behavior contracts, and other helpful worksheets to use during group. The final section is
the student guide. This is the specific workbook for the youth. These can be copied and provided
to the youth.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Impact of Crime groups are held on Mondays and Wednesdays from 3:15pm to 4:15pm and
Tuesdays and Thursdays from 3:15pm to 4:15pm. All documentation is maintained in a binder
kept in the program director's office and in the Evidence-Based System with in the Juvenile
Justice Information System. The binder has the group schedules, youth tracking for schedules,
group sign-in sheets, and internal fidelity monitoring forms. The sign-in sheets and fidelity forms
are organized by cohort.
All youth are eligible to complete the Impact of Crime groups. The groups are closed, meaning
once the group has completed the first lesson, no youth may be added.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Impact of Crime was not able to be observed; therefore this indicator is rated as non-
applicable. The group was scheduled to be observed on two separate occassions. On one of the
dates, the facilitator had to leave due to an appointment and on the second scheduled date the
group was cancelled due to a youth testing positive for covid.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program did not experience turnover during this twelve month review period. Should the
program experience turnover, another trained facilitator will complete the group. The program
does attempt to have the facilitator complete the cohort before leaving employment, if possible.
The program completed five cohorts during the twelve month review period.
The first cohort started on September 18, 2020 and ended on December 15, 2020. The group
started with eight youth and six youth successfully completed the group. One youth did have one-
to-one sessions completed to ensure she was able to complete the group prior to leaving the
program. There was no documentation as to why the two other youth did not complete this group
successfully. The facilitators need to ensure they are documenting when a youth leaves a group
and why on the attendance forms in the appropriate spaces.
The second cohort started December 1, 2020 and ended on April 16, 2020. There was a three
week gap during this cohort due to a covid-19 outbreak. All youth in the group and the facilitator
were placed in quarantine from December 29, 2020 to January 26, 2021. This does not qualify as
an “unnecessary gap in service”. The group started with ten youth and seven youth successfully
completed the group. Two youth left the program early; however, the facilitator conducted
individual group sessions. There were separate sign-in sheets and documentation of the
workbook being completed. The third youth was released early by the court in March and was
unable to complete all of the curriculum.
The third cohort started March 1, 2021 and ended on June 23, 2021. Eleven youth began this
group and eleven youth successfully completed the group.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
The fourth cohort started April 27, 2021 and ended on September 7, 2021. The group started with
twelve youth and nine youth successfully completed the group. During this group one youth was
transferred to another program. Two youth were transferred to DOVE Academy from another
program. The youths' previous program let the program director know the two youth never started
or received IOC groups while there. This email documention was provided for review. The two
youth from the other program began sessions with a facilitator to attempt to get through the
curriculum prior to their release. However, the youth were unable to complete. All documentation
supporting this situation was provided for this review. Another youth was released early from the
program, but was able to successfully complete the group with one-to-one sessions with the
facilitator.
The fifth cohort started July 7, 2021 and ended on November 26, 2021. The group started with ten
youth and ended with seven youth. Two youth were released early and did not successfully
complete the group. One of these youth was an early court-ordered release. The program
provided the order for review. Two other youth received individual sessions and were able to
complete the curriculum prior to their release.
The sixth cohort started October 12, 2021 and is continuing to meet at the time of this review. The
group started with five youth. One youth was added to the group during the third session. The
youth came to the program shortly after the group started and another group was not starting for
quite some time. The facilitator conducted the first two sessions as individual sessions with the
youth to ensure she was on track with the group.
The seventh cohort started November 30, 2021 and is continuing to meet at the time of this
review. The group started with sixth youth. Two youth were added in the third session. These two
youth came to the program on December 3, 2021 and a new group was not going to start. The
two youth had individual sessions conducted and entered the large group on December 7, 2021.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
Another trained facilitator conducts internal fidelity monitoring for the Impact of Crime groups.
There were three facilitators during the twelve month review period.
Twenty-five out of twenty-six internal fidelity monitorings were completed during the twelve month
review period. One was missing due to the facilitator and youth being in quarantine. As groups
were not able to be held during that month, no points will be reduced in this section.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program director addresses any issues or corrective actions with the facilitators for Impact of
Crime groups. The transition specialist conducts internal fidelity monitorings for the groups. The
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
program director reviews and signs-off on the internal fidelity monitoring forms weekly. All forms
have a section to document any facilitation and/or group issues.
After a review of the internal fidelity monitoring forms and speaking with the program director, it
was determined there were no significant issues or corrective actions during the twelve month
review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program director provided all performance evaluations for the Impact of Crime facilitators.
Each of the facilitators had received their annual performance evaluations and there was a
section to address delivery of Impact of Crime groups.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service - Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 16 weeks for this type of service. Of the 42 youth in the
sample, 69% (29 of 42) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 42 youth in the
sample, 93% (39 of 42) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 7 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 28%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 76%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 11 youth
Moderate-High = 17 youth
High = 4 youth
Total Youth in Sample = 42 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 10%
High Score: 0 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 0%
Table 2
High = 4 youth
Total Youth in Sample = 42 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 76% (32 of 42) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 10% (4 of 42) youth scored High-Risk to reoffend, for a score of 0 points.
Note: The latest Community Assessment Tool (CAT) prior to the placement date was used in the
derivation of the risk level score. This CAT provides the best indication of the risk to re-offend
level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 7 25 28%
Totals 61 85 72%
This SPEP report evaluates Impact of Crime, an intervention delivered at DOVE Academy.
44 youth completed the program during the review period. Of those youth, 42 were included in the
sample.
The program scored High for Quality of Service Delivery.
The program earned 6 points for Amount of Service: Duration. Of the 42 total youth sampled, only
41 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 29
received at least the recommended weeks of service. Youth in the sample completed between 12
and 20 weeks of service, with an average of 17 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 42 total youth
sampled, all 42 included dosage in the EBS Module. Of those youth, 39 received at least the
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
recommended hours of service. Youth in the sample completed between 12 and 54 hours of
service, with an average of 32 hours.
The program was awarded 7 available points for Risk Level of Youth Served. This is calculated
using data from the Community Assessment Tool (CAT) assessment. This score reflects youths'
most recent CAT score prior to placement at the program. The program itself has no control over
youths' CAT risk level because the scored assessment was administered prior to the youths'
admission.
Had the percentage of youth served with Moderate-High and High risk to reoffend been higher,
and this report awarded each 25 of the available points for this section, the Program Optimization
Percentage for this service would increase to 93%. As stated previously, the Risk Level of Youth
Served score is not under the control of the program.
RECOMMENDATION(S):
DOVE Academy can maintain their SPEP Quality of Service Delivery score by continuing the
practices in place at the time of this review.
DOVE Academy can optimize their SPEP Amount of Service score by ensuring that dosage for all
youth is recorded accurately in EBS and by ensuring that youth receive the full targeted dosage of
service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

file_name: FY2021-22 DOVE - IOC.pdf