summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, , Florida 32934
Primary Service: Seven Challenges
SPEP Review Date(s): August 18-21, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/14/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Melbourne Center for Personal Growth QI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 18-21, 2015 Lead Reviewer Code: 84
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 2 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 6 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group Attendance Logs
Contract Monitoring Reports # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had four staff who delivered the primary service since the last annual compliance
review. Each of the four staff received training from an authorized Seven Challenges leader. The
training consisted of watching a video and co-facilitating delivery of the curriculum with the leader
for seven hours.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The facilitators utilize the Seven Challenges Manual. The manual provides various approaches
including building trust and support and developing the cognitive/emotional decision making
model with a holistic vision. The three basic components include readings, journals and the
counseling/educational sessions. The sessions use current issues the youth are working on, as
well as life skills, exercises, activities, and problem solving sessions.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The team observed two sessions of the primary service on the same day delivered to two groups
of ten youth. Each trained facilitator presented the same exercise/activity in accordance with the
plan for the day. The youth performed an emotional check-in process prior to the exercise. The
facilitator described the exercise, set up the room, and had youth move to complete the exercise.
The facilitator often brought the youth to focus back to the one of the Seven Challenges
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
throughout the session. The facilitator for each group followed the plan provided to the review
team prior to delivery of the primary service.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
There were no gaps noted in the delivery of the primary service; however, there has been
facilitator turnover since the last annual compliance review. There was no record of transition
documented for each instance of staff leaving and new staff taking over an existing group.
The program can earn 2 points if there is no facilitator turnover impacting a group cycle along with
no gaps in service.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program conducted internal fidelity monitoring of the primary service. There were two
different staff conducting the fidelity monitoring since the last annual compliance review. Both
individuals performing the fidelity monitoring have completed formal training in the primary
service. The fidelity monitoring included the use of checklist specific to the intervention; however,
the documentation provided did not support the internal fidelity monitoring was conducted once a
month per facilitator.
The program can earn 2 points if fidelity monitoring is completed every month for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program has a system in place to provide corrective action for evidence based interventions
following fidelity monitoring. The program created a universal form to document corrective action
for evidence based interventions. The form has a box to check next to the specific intervention
(i.e. Impact of Crime, Seven Challenges, or Aggression Replacement Training), and separate
sections for the facilitator(s) name, date, finding, root cause, process measure/outcome, and
specific actions sections. One corrective action form was initiated on July 20, 2015 with a follow-
up action planned for August 28, 2015. However, there was no separate fidelity monitoring report
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
generated for July 20, 2015 documenting the deficiencies for the identified facilitator. Corrective
action had not been completed as of the date of the SPEP review.
The program can earn 1 point if fidelity monitoring is completed every month for each facilitator
and any required corrective action is documented, demostrated, and applied.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Of the four staff who facilitated groups since the last annual compliance review, three are
currently working at the program. None of the three current facilitators have been on staff long
enough to receive an annual employee evaluation. For the fourth facilitator (who has since left the
program), the annual or probationary evaluation did not evaluate the staff in the provision of the
primary service.
The program can earn 1 point if each facilitator's performance evaluation includes an assessment
of their skill in the delivery of the primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 20 youth in the
sample, 0% (0 of 20) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 10 youth
High = 9 youth
Total Youth in Sample = 20 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 20 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (19 of 20) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 45% (9 of 20) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 65 95 68%
This SPEP report evaluates Seven Challenges, an intervention delivered at Melbourne Center for
Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
minimizing turnover, especially with a group cycle. Internal fidelity monitoring should be done
monthly for each facilitator. Performance evaluations should address each facilitator's skill in
delivering the primary service.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 24 weeks. Of the 20 total youth sampled, only 1 had dosage recorded
in EBS with an end date. That youth completed 4 weeks of service.
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 40 hours. Of the 20 total youth sampled, only 1 had dosage
recorded in EBS. That youth completed 7.5 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by minimizing turnover, completing fidelity monitoring monthly for each facilitator, and ensuring
performance evaluations address each facilitator's skill in delivering the primary service.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

report_type: SPEP

program_status_note: Closed

service_type: Seven Challenges


state: FL

facility_name: melbourne center for personal growth

summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Aggression Replacement Training
SPEP Review Date(s): August 18-21, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/14/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Melbourne Center for Personal Growth QI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 18-21, 2015 Lead Reviewer Code: 84
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee 1 # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group Attendance Logs
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 25 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
All five staff facilitating the primary service have received formal training by a qualified trainer.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Aggression Replacement Training curriculum. The manual contains
lesson plans and handouts and details how the curriculum is to be implemented.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The delivery of the primary service was observed. During the observation, it was noted the
facilitator followed the lesson as outlined in the manual, and completed all activities as required.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The program has had five staff deliver the Aggression Replacement Training curriculum since the
last annual compliance review. The majority of groups within each cohort was facilitated by the
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
same two facilitators. Each cohort has been conducted by the same two facilitators throughout the
cycle. There have been no gaps in service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program conducted internal fidelity monitoring of the primary service six of the past eight
months. The staff performing fidelity monitoring have completed formal training in the primary
service. The fidelity monitoring included the use of checklist specific to the intervention; however,
the documentation provided did not support the internal fidelity monitoring was conducted once a
month per facilitator.
The program can earn 2 points if internal fidelity monitoring is completed every month for each
facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the internal fidelity monitoring documentation found no evidence corrective action was
applicable. However, the program created a universal evidence based corrective action form to
document corrective action if needed. The form has a box to check next to the specific
intervention (i.e. Impact of Crime, Seven Challenges, or Aggression Replacement Training), and
separate sections for the facilitator(s) name, date, finding, root cause, process measure/outcome,
and specific actions sections.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program conducts evaluations at ninety-days and annually. Only one of five facilitators
received an evaluation which included an assessment of the facilitator’s skill in delivering the
primary service.
The program can earn 1 point if each facilitator is evaluated on their skill in the delivery of the
primary service.
At the time of the review, the program did not meet criteria to earn a score of 1.
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 10 weeks for this type of service. Of the 22 youth in the
sample, 0% (0 of 22) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 30 contact hours for this type of service. Of the 22 youth in the
sample, 14% (3 of 22) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 11 youth
High = 10 youth
Total Youth in Sample = 22 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 22 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (21 of 22) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 45% (10 of 22) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 80 100 80%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Melbourne Center for Personal Growth.
The program scored High for Quality of Service Delivery. This score can be optimized by
completing internal fidelity monitoring every month for each facilittaor and including an evaluation
of skill in delivering this primary service in the employee perfomance evaluations.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this specific intervention is 10 weeks. In previous SPEP review years, ART was held to the same
standard as other Group 5 Services (15 weeks); however, according to Dr. Lipsey's SPEP
research, ART meets criteria for an exception to the standard SPEP 15-week dosage target.
Youth are considered to have met the target if they achieve 80% of the dosage specified in the
ART manual. Of the 22 total youth sampled in this SPEP report, only 3 included dosage in the
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
EBS Module and none included end dates, therefore, it was impossible to determine the number
of weeks completed.
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 30 hours. In previous SPEP review years, ART was held to
the same standard as other Group 5 Services (45 hours); however, according to Dr. Lipsey's
SPEP research, ART meets criteria for an exception to the standard SPEP 45-hour dosage target.
Youth are considered to have met the target if they achieve 80% of the dosage specified in the
ART manual. Of the 22 total youth sampled in this SPEP report, only 3 included dosage in the
EBS Module. Of those youth with dosage, all 3 received at least the recommended hours of
service. All youth in the sample completed 29 or 30 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by completing internal fidelity monitoring every month for each facilittaor and including an
evaluation of skill in delivering this primary service in the employee perfomance evaluations
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

source: DJJ

program_name_raw: Melbourne Center for Personal Growth

service_type: Aggression Replacement Training


summary: SPEP - FY15-16

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Impact of Crime: Addressing the Harm to Victims and the
Community
SPEP Review Date(s): August 18-21, 2015
Florida Department of Juvenile Justice
Report Date(s): 10/14/2015
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
2
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Program Name: Melbourne Center for Personal Growth QI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 18-21, 2015 Lead Reviewer Code: 84
Persons Interviewed
Program Director Corporate QI/QA staff # Program Supervisors
DJJ Monitor # Case Managers # Youth
DHA or designee # Clinical Staff # Other (listed by title):
DMHCA or designee # Healthcare Staff
Documents Reviewed
Written Protocol/Manual Logbooks # Personnel Records
Fidelity Monitoring Documents Program Schedules 4 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook 1 # Other: Group Attendance Logs
Contract Monitoring Reports # Health Records
Contract Scope of Services # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
3
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime: Addressing the Harm to Victims and
Communities. The program was awarded 15 points because the Primary Service is identified as
a Group 3 Service. The specific Sub-Component Service Type identified is Social Skills Training.
The Primary Service was identified as this type of service as it focuses on developing social skills
required for an individual to interact in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
4
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
All four staff participating in the delivery of the curriculum and/or responsible for performing fidelity
monitoring have completed formal training.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program utilizes the Impact of Crime seven-chapter curriculum. The manual includes a
protocol for the frequency and duration of the service, an outline and lesson plan for each topic,
as well as an order in which the topics are to be delivered.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The delivery of the primary service was observed. During the observation, it was noted the
facilitator followed the chapter, as outlined in the facilitator guide, and completed all activities as
required. The facilitator also showed videos on ‘You Tube’ relevant to the lesson plan.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
5
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
The program has completed three cohorts of Impact of Crime since the last annual compliance
review, and recently began a fourth cohort. There were no gaps noted in the delivery of the
primary service, however, in two of the cohorts, two different facilitators were documented
facilitating different lessons. The documentation was not clear as to whether this was due to
facilitator turnover in the facility.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The program conducted internal fidelity monitoring of the primary service. The staff performing the
fidelity monitorings have completed formal training in the primary service. The fidelity monitoring
included the use of checklist specific to the intervention; however, the documentation provided did
not support the internal fidelity monitoring was conducted once a month per facilitator. One
facilitator had documentation to support fidelity monitoring for four of seven months in which he
conducted groups and the second facilitator had documentation to support fidelity monitoring for
three of five months in which she conducted groups.
The program can earn 2 points if fidelity monitoring is completed each moth for each facilitator.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
A review of the internal fidelity monitoring documentation found no evidence of corrective action
being needed. However, the program created a universal evidence based corrective action form
to document corrective action if needed. The form has a box to check next to the specific
intervention (i.e. Impact of Crime, Seven Challenges, or Aggression Replacement Training), and
separate sections for the facilitator(s) name, date, finding, root cause, process measure/outcome,
and specific actions sections.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
The program conducts evaluations at ninety-days and annually. One facilitator was applicable for
a ninety day evaluation and one facilitator was applicable for an annual evaluation. Both
facilitators received the applicable evidence based services facilitator evaluation, which included
an assessment of the facilitator’s skill in delivering this primary service.
6
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
7
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
3. Amount of Service - Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 22 youth in the
sample, 0% (0 of 22) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 22 youth in the
sample, 18% (4 of 22) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 0 youth
Moderate-High = 11 youth
High = 10 youth
Total Youth in Sample = 22 youth
8
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
Percentage of Youth with High-Risk Level to Reoffend: 45%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 22 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (21 of 22) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 45% (10 of 22) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 65 85 76%
This SPEP report evaluates Impact of Crime, an intervention delivered at Melbourne Center for
Personal Growth.
The program scored High for Quality of Service Delivery. This score can be optimized by
minimizing facilitator turnover. This includes inconsistency in group delivery. The same facilitator
or co-facilitators should complete the entire cycle when possible. Internal fidelity monitoring
should be completed every month for every facilitator.
The program earned 0 points for Amount of Service: Duration. The targeted number of weeks for
this type of intervention is 16 weeks. Of the 22 total youth sampled, none had dosage with end
dates, therefore, it was impossible to ascertain the number of weeks completed.
9
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report
The program earned 0 points for Amount of Service: Contact Hours. The targeted number of
hours for this type of intervention is 24 hours. Of the 22 total youth sampled, only 4 had dosage
recorded in EBS. Those 4 youth completed exactly 24 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by minimizing facilitator turnover and and completing internal fidelity monitoring every month for
every facilitator.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
10
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report

service_type: Impact of Crime


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Drive
Melbourne, Florida 32934
Primary Service: Seven Challenges
SPEP Review Date(s): August 23‐26, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/5/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 23-26, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director 3 # Clinical Staff director of operations,
DJJ Monitor 2 # Healthcare Staff psychiatrist, compliance manager,
DHA or designee 3 # Program Supervisors Circles of Care Human Resources
DMHCA or designee 5 # Youth representative,
Corporate QI/QA staff 6 # Other (listed by title): director of
1 # Case Managers education, business manager,
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 15 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There were six clinicians facilitating Seven Challenges groups to the youth in the past twelve
months. There was documentation to support all six received the facilitator training.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for reference during
the annual compliance review. The program currently uses the Seven Challenges. The curriculum
provides a manualized direction for the facilitator.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A session of Seven Challenges was observed during the annual compliance review. Prior to the
observation of the Seven Challenges group, the facilitator provided the reviewer with a copy of the
lesson from the manual, as well as a copy of the handouts to be provided to the youth. The lesson
focused on visual learning skills and the youth were able to connect this lesson to recent lessons
on listening and language skills. Each youth was tasked with using a phrase, and describing it
with pictures, not words. The youth were fully engaged in the activity and each was able to
present his drawing to the group. The group was facilitated in the dining hall, which presented
enough room for the youth and the facilitator. The group started with each person, including the
youth, facilitator and direct care staff, offering expectations and a check in on how their day was
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
going. Following the presentation, the youth were able to state which of the Seven Challenges the
exercise pertained to.
Basic Score: 1 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Two of the clinicians trained to provide Seven Challenges are no longer employed at the program.
There was documentation and interviews with the clinical director to support there were no gaps
in the delivery of the intervention in the last twelve months. There is no written protocol or policy
on provision of services in case of turnover. All youth are scheduled to complete Seven
Challenges prior to their release from the program.
The program can earn 2 points by minimizing facilitator turnover.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was documentation of fidelity monitoring completed monthly for each of the past twelve
months and the fidelity monitoring was consistently completed on the Seven Challenges
Counselor Skills form. The program does not have a written protocol requiring monthly fidelity
monitoring. All of the fidelity monitoring was conducted by a staff trained in the Seven Challenges
curricula. All checklists were signed by the fidelity monitor and the staff being monitored.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was documentation of fidelity monitoring completed monthly each of the past twelve
months. The fidelity monitoring was consistently completed on the Department of Juvenile Justice
fidelity adherence checklist for Seven Challenges. There was documentation in the form of a
corrective action plan regarding issues noted during the fidelity monitoring. The findings were
discussed with the applicable facilitator in April 2016. The form included documentation the
findings were discussed with the clinician, and a reassessment would be completed the following
month, which was completed. The program does not have a written protocol or process describing
in-house corrective action in case it was indicated.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the facilitators employee performance evaluations found no evaluation of the staff’s
performance in delivery of Seven Challenges. The program does not have a written protocol or
process that includes the requirement for evaluation of the facilitator’s performance in delivery of
the primary service.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of Seven Challenges.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 24 weeks for this type of service. Of the 18 youth in the
sample, 61% (11 of 18) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 18 youth in the
sample, 100% (18 of 18) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 83%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 6 youth
High = 9 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 83% (15 of 18) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 50% (9 of 18) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 86 95 91%
This SPEP report evaluates Seven Challenges, an intervention delivered at Melbourne Center for
Personal Growth.
The program scored High for Quality of Service Delivery. This score can be optimized by
minimizing facilitator turnover and by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of Seven Challenges.
The program earned 6 points for Amount of Service: Duration. Of the 18 total youth sampled, 11
received at least the recommended weeks of service. Youth in the sample completed between 4
and 28 weeks of service, with an average of 23 weeks.
The program earned 10 points for Amount of Service: Contact Hours. Of the 18 total youth
sampled, all 18 received at least the recommended hours of service. Youth in the sample
completed between 49 and 89 hours of service, with an average of 66 hours.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by minimizing facilitator turnover and by ensuring that facilitators' performance evaluations include
an assessment of their skill in the delivery of Seven Challenges.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Drive
Melbourne, Florida 32934
Primary Service: Aggression Replacement Training
SPEP Review Date(s): August 23‐26, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/5/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 23-26, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director 3 # Clinical Staff director of operations,
DJJ Monitor 2 # Healthcare Staff psychiatrist, compliance manager,
DHA or designee 3 # Program Supervisors Circles of Care Human Resources
DMHCA or designee 5 # Youth representative
Corporate QI/QA staff 6 # Other (listed by title): director of
1 # Case Managers education, business manager,
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 15 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training (ART). The program
was awarded 30 points because the Primary Service is identified as a Group 5 Service. The
specific Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary
Service was identified as this type of service as it is intended to correct faulty cognitions and
perceptions and provides skills individuals can use to monitor thought patterns and correct
behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
There were four clinicians and the director of case management providing Aggression
Replacement Training (ART) groups to the youth in the past twelve months. There was
documentation to support three clinicians and the director of case management received the
facilitator training. The program’s clinical director and a second clinician were trained on the
primary service in April 2016; the program did not receive documentation of the training received
by these two clinicians. There was an email sent to the provider’s training coordinator in an
attempt to gain certification of the training; however, this documentation was not provided to the
reviewers at the time of the review.
The program can earn 1 point by ensuring that all facilitators are trained to deliver ART.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for reference during
the annual compliance review. The program currently uses the Aggression Replacement Training
(ART) curriculum. The curriculum provides a manualized direction for the facilitator.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
Two of the four clinicians trained to provide Aggression Replacement Training (ART) are no
longer employed at the program. There was documentation and interviews with the clinical
director to support there were no gaps in the delivery of the intervention in the last twelve months.
ART groups continued to be facilitated by the other two clinicians. There is no written protocol or
policy on provision of services in case of turnover. All youth are scheduled to complete ART prior
to their release from the program.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
There was documentation of fidelity monitoring completed monthly for each of the past twelve
months; the fidelity monitoring was consistently completed on the Department of Juvenile Justice
fidelity adherence checklist for Aggression Replacement Training (ART). The program does not
have a written protocol requiring monthly fidelity monitoring. All of the fidelity monitoring was
conducted by a staff training in the ART curricula. All checklists were signed by the fidelity monitor
and the staff being monitored.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
There was documentation of fidelity monitoring completed monthly each of the past twelve
months; the fidelity monitoring was consistently completed on the Department of Juvenile Justice
fidelity adherence checklist for Aggression Replacement Training. There was documentation in
the form of a corrective action plan regarding issues noted during the fidelity monitoring. The
findings were discussed with the applicable facilitator in May 2016. The clinician resigned from the
program shortly thereafter, due to a family emergency, therefore did not conduct ART groups after
that point.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of the facilitators employee performance evaluations found no evaluation of the staff’s
performance in delivery of Aggression Replacement Training. The program does not have a
written protocol or process that includes the requirement for evaluation of the facilitator’s
performance in delivery of the primary service.
The program can earn 1 point by ensuring that all facilitators' performance evaluations contain an
assessment of their performance in the delivery of ART.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 16 youth in the
sample, 100% (16 of 16) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 16 youth in the
sample, 100% (16 of 16) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 81%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 5 youth
High = 8 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 81% (13 of 16) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 50% (8 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 20 25 80%
Totals 85 100 85%
This SPEP report evaluates Aggression Replacement Training (ART), an intervention delivered at
Melbourne Center for Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring that all facilitators are trained to deliver ART and by ensuring that all facilitators'
performance evaluations contain an assessment of their performance in the delivery of ART.
The program earned 10 points for Amount of Service: Duration. Of the 16 total youth sampled, all
youth received at least the recommended weeks of service.
The program earned 10 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, all received at least the recommended hours of service.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by by ensuring that all facilitators are trained to deliver ART and by ensuring that all facilitators'
performance evaluations contain an assessment of their performance in the delivery of ART.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Drive
Melbourne, Florida 32934
Primary Service: The Council For Boys and Young Men (Boys Council)
SPEP Review Date(s): August 23‐26, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/5/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 23-26, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director 3 # Clinical Staff director of operations,
DJJ Monitor 2 # Healthcare Staff psychiatrist, compliance manager,
DHA or designee 3 # Program Supervisors Circles of Care Human Resources
DMHCA or designee 5 # Youth representative,
Corporate QI/QA staff 6 # Other (listed by title): director of
1 # Case Managers education, business manager,
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 15 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is The Council For Boys and Young Men (Boys Council).
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 6 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program documentation included certification of four staff completing the training certification
by the One Circle Foundation. Three of the trained staff received the qualified training in previous
years. One staff completed the training the month prior to the review. One trained facilitator has
been the primary staff delivering the Boys Council for the last twelve months. An interview with the
facilitator revealed the other trained staff perform fidelity monitoring. The program does not have a
written training protocol including guidelines for delivery of the Boys Council.
The program can earn 1 point by ensuring that there is a written training protocol.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for reference during
the review. The program currently uses the Living a Legacy: A Rite of Passage (Boys Council)
copyright 2012. The curriculum provides an outline for each of the ten sessions including, activity,
materials, purpose, facilitator preparation guidelines. Each lesson includes the following sections:
Opening ritual, Theme Introduction, Warm-up Activity, Check-in, Activity, Reflection, and Closing
Ritual. There was also a Leadership Quote for each of the ten lessons. The curriculum is not
designed for each youth to have an individual manual. However, there are handouts, worksheets,
or exercises designed for each activity in the lessons.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
A member of the review team observed a session of Boys Council during the review week. The
facilitator provided the team prior to delivery of the intervention a copy of lesson 9: No One Walks
Alone: Leading the way for Younger Boys. The facilitator delivered the intervention at the regular
scheduled time in a group room accommodating the ten youth and two observers. There were no
interruptions during the group. The facilitator followed the lesson according to the manual, not
reading the script verbatim, but delivered in an easy manner. The facilitator appeared well
acquainted with the text and process of the lesson throughout, including transitions from section
to section of the lesson plan.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and interviews with the facilitator, clinical director, and director of
operations revealed there were no gaps in service and no gaps in delivery of the intervention in
the last twelve months. There is no written protocol or policy on provision of services in case of
turnover. All five closed reviewed files revealed each youth participated in the Boys Council during
their time in the program.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The team reviewed twelve months of internal fidelity monitoring reports. There was documentation
of fidelity monitoring completed monthly since March 2016, and in January of 2016. However,
there was no documentation of internal fidelity monitoring from August 2015 to December 2015.
The program does not have a written protocol requiring monthly fidelity monitoring. Three different
staff provided the internal fidelity monitoring. Each of the three staff had been trained by a
qualified trainer in delivery of the intervention and used an official form designed by the publisher,
One Circle Foundation for the purpose of quality assurance.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly for each facilitator of The Council.
At the time of the review, the program did not meet criteria to earn a score of 2.
BasicScore: 0Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
The program documentation of internal fidelity monitoring reports did not reveal any indication
corrective action was required. An interview with the clinical director and the primary facilitator
revealed no corrective action had been required in the last twelve months. The program is using
an instrument specifically designed by the publisher for the purpose of internal fidelity monitoring.
The program does not have a written protocol or process describing in-house corrective action in
case it was indicated.
The program can earn 1 point by ensuring that there is a written internal protocol for corrective
action based on fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the primary facilitator’s employee performance evaluations found no entry for
evaluation of the staff’s performance in delivery of the Boys Council The program does not have a
written protocol or process that includes the requirement for evaluation of the facilitator’s
performance in delivery of the primary service.
The program can earn 1 point by ensuring that facilitators' performance evaluation includes an
assessment of their delivery of The Council.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 16 youth in the
sample, 0% (0 of 16) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 40 contact hours for this type of service. Of the 16 youth in the
sample, 0% (0 of 16) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 81%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 5 youth
High = 8 youth
Total Youth in Sample = 16 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 50%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 8 youth
Total Youth in Sample = 16 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 81% (13 of 16) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 50% (8 of 16) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 20 25 80%
Totals 60 95 63%
This SPEP report evaluates The Council For Boys and Young Men (Boys Council), an
intervention delivered at Melbourne Center for Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring that there is a written training protocol, by ensuring that internal fidelity monitoring is
conducted at least monthly for each facilitator of The Council, by ensuring that there is a written
internal protocol for corrective action based on fidelity monitoring, and by ensuring that facilitators'
performance evaluation includes an assessment of their delivery of The Council.
The program earned 0 points for Amount of Service: Duration. Of the 16 total youth sampled,
none received at least the recommended weeks of service. Youth in the sample completed
between 7 and 10 weeks of service, with an average of 9 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 16 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 5 and 15 hours of service, with an average of 14 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by ensuring that there is a written training protocol, by ensuring that internal fidelity monitoring is
conducted at least monthly for each facilitator of The Council, by ensuring that there is a written
internal protocol for corrective action based on fidelity monitoring, and by ensuring that facilitators'
performance evaluation includes an assessment of their delivery of The Council.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: The Council for Boys and Young Men