summary: SPEP - FY16-17

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Drive
Melbourne, Florida 32934
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): August 23‐26, 2016
Florida Department of Juvenile Justice
Report Date(s): 6/5/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 30
Review Date(s): August 23-26, 2016 Lead Reviewer Code: 97
Persons Interviewed
Program Director 3 # Clinical Staff director of operations,
DJJ Monitor 2 # Healthcare Staff psychiatrist, compliance manager,
DHA or designee 3 # Program Supervisors Circles of Care Human Resources
DMHCA or designee 5 # Youth representative,
Corporate QI/QA staff 6 # Other (listed by title): director of
1 # Case Managers education, business manager,
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 15 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 5 # Youth Records (Closed)
Staff Evaluations Table of Organization 5 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 5 # Health Records
Contract Scope of Services 5 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 7 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program documentation included certification of three staff completing the training
certification by the qualified trainers. Two trained facilitators have been the primary staff delivering
the Impact of Crime (IOC) curriculum. An interview with the facilitator and the clinical director
revealed a facilitator trained by a qualified trainer in IOC comes on campus monthly and provides
fidelity monitoring for both IOC facilitators. The program does not have a written training protocol
including guidelines for training in the delivery and fidelity monitoring of IOC.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for reference during
the review. The program currently uses the Department’s Impact of Crime (IOC): Addressing the
Harm to Victims and the Community. The IOC used is the revised curriculum that was updated in
2011. The revised IOC is a victim impact/restorative justice curriculum consisting of seven
chapters taking approximately twenty-eight sessions to facilitate. The curriculum provides a
manualized direction for the facilitator.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
This curriculum was not able to be observed; therefore, this indicator rates as non-applicable.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
The reviewed documentation and interviews with the facilitator, clinical director, and director of
operations revealed there were no gaps in service and no gaps in delivery of the intervention in
the last twelve months. There is no written protocol or policy on provision of services in case of
turnover. All youth completed IOC prior to release.
BasicScore: 2Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The team reviewed twelve months of internal fidelity monitoring reports. There was documentation
of fidelity monitoring completed monthly for each of the past twelve months. The program does
not have a written protocol requiring monthly fidelity monitoring. One staff trained by a qualified
trainer has been identified to provide the monthly internal fidelity monitoring. All twelve internal
fidelity monitoring events were documented on the official IOC Fidelity Adherence Checklist. All
checklists were signed by the fidelity monitor and the staff being monitored.
BasicScore: 0Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
The program documentation of internal fidelity monitoring reports did not reveal any indication
corrective action was required. An interview with the clinical director and the primary facilitator
revealed no corrective action had been required in the last twelve months. The program is using
the official Impact of Crime (IOC) Fidelity Adherence Checklist. The program does not have a
written protocol or process describing in-house corrective action in case it was indicated.
The program can earn 1 point by ensuring that the program has a written protocol for corrective
action based on fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of the primary facilitator’s employee performance evaluations found no entry for
evaluation of the staff’s performance in delivery of the Impact of Crime. The program does not
have a written protocol or process that includes the requirement for evaluation of the facilitator’s
performance in delivery of the primary service.
The program can earn 1 point by ensuring that facilitators' performance evaluations contain an
assessment of their delivery of IOC.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 4 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 40%
Research indicates the target duration of 16 weeks for this type of service. Of the 17 youth in the
sample, 41% (7 of 17) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 24 contact hours for this type of service. Of the 17 youth in the
sample, 82% (14 of 17) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 20 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 80%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 82%
Moderate to High Score: 7 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 58%
Table 1
Moderate = 0 youth
Moderate-High = 5 youth
High = 9 youth
Total Youth in Sample = 17 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 9 youth
Total Youth in Sample = 17 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 82% (14 of 17) youth scored Moderate to High-Risk to reoffend, for a score
of 7 points.
Of the SPEP sample, 53% (9 of 17) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 4 10 40%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 20 25 80%
Totals 62 85 73%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Melbourne Center
for Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring that the program has a written protocol for internal fidelity monitoring, by ensuring that
the program has a written protocol for corrective action based on fidelity monitoring, and by
ensuring that facilitators' performance evaluations contain an assessment of their delivery of IOC.
The program earned 4 points for Amount of Service: Duration. Of the 17 total youth sampled, 7
received at least the recommended weeks of service. Youth in the sample completed between 13
and 18 weeks of service, with an average of 15 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 17 total youth
sampled, 14 received at least the recommended hours of service. Youth in the sample completed
between 18 and 24 hours of service, with an average of 23 hours.
The program was awarded 20 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by ensuring that the program has a written protocol for internal fidelity monitoring, by ensuring that
the program has a written protocol for corrective action based on fidelity monitoring, and by
ensuring that facilitators' performance evaluations contain an assessment of their delivery of IOC.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10


state: FL

facility_name: melbourne center for personal growth

summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Seven Challenges
SPEP Review Date(s): July 18‐21, 2017
Florida Department of Juvenile Justice
Report Date(s): 10/11/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 32
Review Date(s): July 18-21, 2017 Lead Reviewer Code: 155
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 1 # Case Managers 7 # Youth
DHA or designee 2 # Clinical Staff 7 # Other (listed by title): Direct
DMHCA or designee # Healthcare Staff Care Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Seven Challenges. The program was awarded 25 points
because the Primary Service is identified as a Group 4 Service. The specific Sub-Component
Service Type identified is Group Counseling. The Primary Service was identified as this type of
service as it focuses on psychological or interpersonal problems or issues faced by an individual
and involves a group of youths interacting with each other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 4 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had documentation to reflect five mental health staff completed the training protocol
and were certified to facilitate the Seven Challenges curriculum during the annual compliance
review period. All five staff watched the required three-hour orientation video and were observed
facilitating groups by a certified Train-the-Trainer mental health staff.
Basic Score: 0 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
There is a specific curriculum for Seven Challenges. When the director of case management was
asked if the mental health staff used a specific curriculum during groups, she indicated groups are
lead by the youth’s conversation, so groups are not lead by a specific curriculum. When the
annual compliance review team member observed a Seven Challenges group, the mental health
staff had a specific lesson planned for the group, but informed the review team member the youth
had a very good conversation the day before and if the youth wanted to continue during this group
she would let them continue it.
The program can earn 2 points by utilizing the published Seven Challenges curriculum.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
During the annual compliance review, three Seven Challenges groups were observed by annual
compliance review team members. The group facilitators provided a copy of the lesson plan to the
observers. The facilitator followed the lesson plan; however, the lesson plan did not come from
the manual.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the group notes and an interview with the director of case management confirmed
there has been no turnover which adversely affected the youth since the last annual compliance
review. The program conducted three different groups of Seven Challenges simultaneously
throughout the review period. For two of those groups, documentation supported one mental
health staff has facilitated the group since the last group. For the third group, documentation
supports one mental health staff has consistently facilitated the group throughout the review
period; one mental health staff facilitated the group at the beginning of the review period then left
the program; the director of case management facilitated some of the groups during the review
period.
BasicScore: 1Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The annual compliance review team reviewed eleven months of fidelity monitoring. The program
has a process of completing internal fidelity monitoring specific to Seven Challenges. At the
beginning of the review period, the designated mental health authority conducted fidelity
monitoring for the four staff facilitating the groups. Once the designated mental health authority
left the program, the director of case management completed fidelity monitoring on staff. Fidelity
monitoring was completed on all staff on a monthly basis except for when the director of case
manager was conducting groups. There was no fidelity monitoring conducted on the director of
case management. None of the mental health staff required corrective action based on the
findings of the fidelity monitoring.
The program can earn 2 points by ensuring that internal fidelity monitoring is conducted at least
monthly for all facilitators.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
An interview with the facility administrator and director of case management was conducted to
gain insight into the process by which corrective action is applied based on fidelity monitoring. The
program does not have an internal process to develop corrective action for staff based on
information gathered from fidelity monitoring.
The program can earn 1 point by developing an internal process for corrective action based on
fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the mental health staff’s employee performance evaluations found there was no area
to address the evaluation of skill in delivering the intervention.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of Seven Challenges.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 95% (18 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 95% (18 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 8 10 80%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 81 95 85%
This SPEP report evaluates Seven Challenges, an intervention delivered at Melbourne Center for
Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
utilizing the published Seven Challenges curriculum, by ensuring that internal fidelity monitoring is
conducted at least monthly for all facilitators, by developing an internal process for corrective
action based on fidelity monitoring and by ensuring that facilitators' performance evaluations
include an assessment of their skill in the delivery of Seven Challenges.
The program earned 8 points for Amount of Service: Duration. Of the 19 total youth sampled, 18
received at least the recommended weeks of service. Youth completed between 24 and 53 weeks
of service with an average of 32 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 8 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, 18 received at least the recommended hours of service. Youth in the sample completed
between 1 and 121 hours of service with an average of 73 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by utilizing the published Seven Challenges curriculum during all groups, by ensuring that internal
fidelity monitoring is conducted at least monthly for all facilitators, by developing an internal
process for corrective action based on fidelity monitoring and by ensuring that facilitators'
performance evaluations include an assessment of their skill in the delivery of Seven Challenges.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

source: DJJ

report_type: SPEP

program_name_raw: Melbourne Center for Personal Growth

program_status_note: Closed

service_type: Seven Challenges


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Aggression Replacement Training (ART)
SPEP Review Date(s): July 18‐21, 2017
Florida Department of Juvenile Justice
Report Date(s): 10/11/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 32
Review Date(s): July 18-21, 2017 Lead Reviewer Code: 155
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 1 # Case Managers 7 # Youth
DHA or designee 2 # Clinical Staff 7 # Other (listed by title): Direct
DMHCA or designee # Healthcare Staff Care Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training (ART). The program
was awarded 30 points because the Primary Service is identified as a Group 5 Service. The
specific Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary
Service was identified as this type of service as it is intended to correct faulty cognitions and
perceptions and provides skills individuals can use to monitor thought patterns and correct
behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had documentation to reflect four mental health staff and the director of case
management completed the training protocol and were certified to facilitate the Aggression
Replacement Training (ART) curriculum during the annual compliance review period. All five staff
completed the two to three-day clinical training. Group notes indicated three therapists facilitated
ART groups during the annual compliance review period.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
There is a specific curriculum for Aggression Replacement Training (ART). When group is being
facilitated, the program utilizes the specific Aggression Replacement Training curriculum.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program was not facilitating the Agression Replacement Training group during the annual
compliance review week. The group was scheduled to begin on July 24, 2017.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the group notes and an interview with the director of case management confirmed
there has been no staff turnover which adversely affected the youth since the last annual
compliance review. The program conducted five cohorts during the review period. Each cohort
was conducted by two facilitators. During one of the cohorts, one of the facilitators left the
program; however, the other facilitator continued to run the group. There were no gaps in service.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The annual compliance review team reviewed eleven months of fidelity monitoring. The program
has a process of completing internal fidelity monitoring specific to Aggression Replacement
Training. Throughout the review period, the director of case management conducted fidelity
monitoring for the four staff facilitating the groups. Fidelity monitoring was completed on all staff
on a monthly basis. None of the mental health staff required corrective action based on the
findings of the fidelity monitoring.
Basic Score: 1 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the facility administrator and director of case management was conducted to
gain insight into the process by which corrective action is applied based on fidelity monitoring. The
program does not have an internal process to develop corrective action for staff based on
information gathered from fidelity monitoring.
The program can earn 1 point by developing an internal process for corrective action based on
fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of the mental health staff’s employee performance evaluations found there was no area
to address the evaluation of skill in delivering the intervention.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of ART.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates the target duration of 10 weeks for this type of service. Of the 19 youth in the
sample, 100% (19 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 10 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 100%
Research indicates a target of 30 contact hours for this type of service. Of the 19 youth in the
sample, 100% (19 of 19) reached the indicated target contact hours. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 18 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 19 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 10 10 100%
Amount of Service: Contact Hours 10 10 100%
Risk Level of Youth Served 25 25 100%
Totals 100 100 100%
This SPEP report evaluates Aggression Replacement Training (ART), an intervention delivered at
Melbourne Center for Personal Growth.
The program scored High for Quality of Service Delivery. This score can be optimized by
developing an internal process for corrective action based on fidelity monitoring and by ensuring
that facilitators' performance evaluations include an assessment of their skill in the delivery of
ART.
The program earned 10 points for Amount of Service: Duration. Of the 19 total youth sampled, all
19 received at least the recommended weeks of service. All youth in the sample completed
approximately 10 weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 10 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, all 19 received at least the recommended hours of service. All youth in the sample
completed 30 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by developing an internal process for corrective action based on fidelity monitoring and by
ensuring that facilitators' performance evaluations include an assessment of their skill in the
delivery of ART.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Aggression Replacement Training


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: The Council For Boys and Young Men (Boys Council)
SPEP Review Date(s): July 18‐21, 2017
Florida Department of Juvenile Justice
Report Date(s): 10/11/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 32
Review Date(s): July 18-21, 2017 Lead Reviewer Code: 155
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 1 # Case Managers 7 # Youth
DHA or designee 2 # Clinical Staff 7 # Other (listed by title): Direct
DMHCA or designee # Healthcare Staff Care Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 30 Points
1. Primary Service and Supplemental Service Types POS: 30 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is The Council For Boys and Young Men (Boys Council).
The program was awarded 25 points because the Primary Service is identified as a Group 4
Service. The specific Sub-Component Service Type identified is Group Counseling. The Primary
Service was identified as this type of service as it focuses on psychological or interpersonal
problems or issues faced by an individual and involves a group of youths interacting with each
other.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 10 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 50%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 5 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 0 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had documentation to reflect two mental health staff completed the training protocol
and were certified to facilitate the The Council for Boys and Young Men curriculum; however, a
third mental health staff facilitated the group during the annual compliance review period. The
program was unable to provide a training certificate for this staff.
The program can earn 1 point by ensuring all facilitators receive training.
At the time of the review, the program did not meet criteria to earn a score of 1.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program used three distinct Council curriculum guides: Standing Together: A Journey into
Respect, Growing Healthy, Growing Strong, and Living, a Legacy: The Council Rite of Passage.
The curricula are detailed and include a specific outline for the session. Group notes documented
the lesson discussed each week.
Basic Score: N/A Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The program did not conduct The Council for Boys and Young Men group while the annual
compliance review team was on-site; therefore, the group was unable to be observed.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of group notes and an interview with the director of case management confirmed there
has been no turnover which adversely affected the youth since the last annual compliance review.
The program conducted six different cohorts of The Council for Boys and Young Men during the
review period. Documentation supports one mental health staff facilitated the group since the last
review. There were no gaps in service.
Basic Score: 1 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The annual compliance review team reviewed eleven months of fidelity monitoring. The program
had a process of completing internal fidelity monitoring specific to The Council for Boys and
Young Men. The five staff took turns oberserving one another facilitate groups. The program was
able to provide a certificate of training for two of the five staff conducting fidelity monitoring. None
of the staff required corrective action based on the findings of the fidelity monitoring.
The program can earn 2 points by ensuring that all individuals conducting internal fidelity
monitoring have been trained in the intervention.
At the time of the review, the program did not meet criteria to earn a score of 2.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the facility administrator and director of case management was conducted to
gain insight into the process by which corrective action is applied based on fidelity monitoring. The
program does not have an internal process to develop corrective action for staff based on
information gathered from fidelity monitoring.
The program can earn 1 point by developing an internal process for corrective action based on
fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of the mental health staff’s employee performance evaluations found there was no area
to address the evaluation of skill in delivering the intervention.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of ART.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 24 weeks for this type of service. Of the 19 youth in the
sample, 0% (0 of 19) reached at least the indicated target duration. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 40 contact hours for this type of service. Of the 19 youth in the
sample, 0% (0 of 19) reached the indicated target contact hours. Further explanation is detailed in
the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 30 30 100%
Quality of Service Delivery 10 20 50%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 0 10 0%
Risk Level of Youth Served 25 25 100%
Totals 65 95 68%
This SPEP report evaluates The Council For Boys and Young Men (Boys Council), an
intervention delivered at Melbourne Center for Personal Growth.
The program scored Medium for Quality of Service Delivery. This score can be optimized by
ensuring all facilitators receive training, by ensuring that all individuals conducting internal fidelity
monitoring have been trained in the intervention, by developing an internal process for corrective
action based on fidelity monitoring, and by ensuring that facilitators' performance evaluations
include an assessment of their skill in the delivery of The Council.
The program earned 0 points for Amount of Service: Duration. Of the 19 total youth sampled,
none received at least the recommended weeks of service. Youth completed between 9 and 10
weeks of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, none received at least the recommended hours of service. Youth in the sample
completed between 12 and 15 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by ensuring all facilitators receive training, by ensuring that all individuals conducting internal
fidelity monitoring have been trained in the intervention, by developing an internal process for
corrective action based on fidelity monitoring, and by ensuring that facilitators' performance
evaluations include an assessment of their skill in the delivery of The Council.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: The Council for Boys and Young Men


summary: SPEP - FY17-18

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMIkids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Impact of Crime (IOC)
SPEP Review Date(s): July 18‐21, 2017
Florida Department of Juvenile Justice
Report Date(s): 10/11/2017
Introduction
The Standardized Program Evaluation Protocol (SPEP) is an assessment tool derived from
meta-analytic research on the effectiveness of juvenile justice interventions. The tool is
designed to compare existing intervention services, as implemented in the field, to the
characteristics of the most effective intervention services found in the research.
The SPEP scoring system allows service providers to identify specific areas in which program
improvements can be made to their existing Primary Services. These improvements can be
expected to increase the effectiveness of those Primary Services in the reduction of recidivism
for youth receiving the Primary Service. A separate SPEP evaluation is conducted, at the time
of the program’s Quality Improvement Review, for each Primary Service provided by the
program.
This report provides two types of SPEP scores: a Basic Score, equivalent to the number of
points received, and a Program Optimization Score (POS) that is equivalent to the maximum
number of possible points that could be received based on the SPEP domains under the control
of the program. The Basic Score compares the Primary Service being evaluated to other
intervention services found in the research to be effective, regardless of service type. It is meant
as a reference to the expected overall recidivism reduction when compared to other Primary
Services of any Type.
A Program Optimization Percentage (POP) rate is derived from the Basic Score and Program
Optimization Score. The POP rate is a percentage score that indicates where the rate of
effectiveness of the Primary Service is when compared to its potential effectiveness if optimized
to match the characteristics of similar Primary Services found to be most effective in the
research. The POP rate is likely more meaningful to service providers as it represents how
close the program’s Primary Service is to its potential for that Primary Service Type. For
example, a POP rate of 55% would indicate that the program’s Primary Service is operating at
55% of its potential effectiveness for recidivism reduction that has been found for a similar
Primary Service Type with research evidence of effectiveness.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Program Name: Melbourne Center for Personal Growth MQI Program Code: 1270
Provider Name: AMIkids, Inc. Contract Number: R2119
Location: Brevard County / Circuit: 18 Number of Beds: 32
Review Date(s): July 18-21, 2017 Lead Reviewer Code: 155
Persons Interviewed
Program Director Corporate QI/QA staff 1 # Program Supervisors
DJJ Monitor 1 # Case Managers 7 # Youth
DHA or designee 2 # Clinical Staff 7 # Other (listed by title): Direct
DMHCA or designee # Healthcare Staff Care Staff
Documents Reviewed
Written Protocol/Manual Logbooks 5 # Personnel Records
Fidelity Monitoring Documents Program Schedules 5 # Training Records/CORE
Internal Corrective Action Reports Supplemental Contracts 3 # Youth Records (Closed)
Staff Evaluations Table of Organization 7 # Youth Records (Open)
Accreditation Reports Youth Handbook # Other:
Contract Monitoring Reports 7 # Health Records
Contract Scope of Services 7 # MH/SA Records
Observations During Review
Group/Session of Primary Service(s)
Program Activities
Recreation
Social Skill Modeling by Staff
Staff Interactions with Youth
Staff Supervision of Youth
Transition/Exit Conferences
Treatment Team Meetings
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
1. Primary Service and Supplemental Service Types POS: 20 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Impact of Crime (IOC). The program was awarded 15
points because the Primary Service is identified as a Group 3 Service. The specific Sub-
Component Service Type identified is Social Skills Training. The Primary Service was identified
as this type of service as it focuses on developing social skills required for an individual to interact
in a positive way with others.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated by the Bureau of Monitoring and Quality Improvement while
on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 8 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
The program had documentation to reflect four mental health staff completed the training protocol
and were certified to train the Impact of Crime curriculum during the annual compliance review
period. All four staff completed the twenty-eight hour training facilitated by the Department. One
therapist became a facilitator in April 2013. Another therapist became a facilitator in November
2016 and the third therapist in January 2017. One therapist became a facilitator in April 2006 and
completed the training again in December 2011. During the review period, three therapists
facilitated the Impact of Crime groups.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The program has a manual for the delivery of services and provided a copy for the team for
reference during the review. The program currently uses Impact for Change: Addressing the Harm
to Victims and the Community. The curriculum provides a script and instructions for seven
chapters. The curriculum provides guidelines for the order the lessons should be delivered and
includes instructions on how the lessons should be delivered.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
During the annual compliance review, a review team member was able to observe an Impact of
Crime group. The group facilitator provided a copy of the lesson plan to the observer. There were
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
seven youth and one facilitator in the group. The facilitator followed the lesson plan according to
the manual and used his own words to deliver the meaning of the lesson in an easy to follow
manner. The facilitator was able to move the youth back on task when the youth became
distracted.
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of group notes and an interview with the director of case management confirmed there
has been no turnover which adversely affected the youth since the last annual compliance review.
The program has conducted a steady flow of Impact of Crime groups throughout the review
period. Three thrapists have consistently facilitated the groups throughout the review period.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The annual compliance review team reviewed eleven months of fidelity monitoring. The program
has a process of completing internal fidelity monitoring specific to Impact of Crime. The three
therapists conduct fidelity monitoring on each other on a monthly basis. None of the mental health
staff required corrective action based on the findings of the fidelity monitoring.
Basic Score: 0 Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the facility administrator and director of case management was conducted to
gain insight into the process by which corrective action is applied based on fidelity monitoring. The
program does not have an internal process to develop corrective action for staff based on
information gathered from fidelity monitoring.
The program can earn 1 point by developing an internal process for corrective action based on
fidelity monitoring.
At the time of the review, the program did not meet criteria to earn a score of 1.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 0 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
A review of the mental health staff’s employee performance evaluations found there was no area
to address the evaluation of skill in delivering the intervention.
The program can earn 1 point by ensuring that facilitators' performance evaluations include an
assessment of their skill in the delivery of IOC.
At the time of the review, the program did not meet criteria to earn a score of 1.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
3. Amount of Service ‐ Duration
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates the target duration of 16 weeks for this type of service. Of the 19 youth in the
sample, 16% (3 of 19) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 0 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 0%
Research indicates a target of 24 contact hours for this type of service. Of the 19 youth in the
sample, 63% (12 of 19) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 95%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 3 youth
Moderate-High = 5 youth
High = 10 youth
Total Youth in Sample = 19 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8
Percentage of Youth with High-Risk Level to Reoffend: 53%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 10 youth
Total Youth in Sample = 18 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 95% (18 of 19) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 53% (10 of 19) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 20 20 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 0 10 0%
Amount of Service: Contact Hours 6 10 60%
Risk Level of Youth Served 25 25 100%
Totals 71 85 84%
This SPEP report evaluates Impact of Crime (IOC), an intervention delivered at Melbourne Center
for Personal Growth.
The program scored High for Quality of Service Delivery. This score can be optimized by
developing an internal process for corrective action based on fidelity monitoring, and by ensuring
that facilitators' performance evaluations include an assessment of their skill in the delivery of
IOC.
The program earned 0 points for Amount of Service: Duration. Of the 19 total youth sampled, 3
received at least the recommended weeks of service. Youth completed between 14 and 21 weeks
of service with an average of 16 weeks.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 9
The program earned 0 points for Amount of Service: Contact Hours. Of the 19 total youth
sampled, 12 received at least the recommended hours of service. Youth in the sample completed
between 23 and 24 hours of service.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can optimize their SPEP Quality of Service Delivery score
by developing an internal process for corrective action based on fidelity monitoring, and by
ensuring that facilitators' performance evaluations include an assessment of their skill in the
delivery of IOC.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that youth receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 10

service_type: Impact of Crime


summary: SPEP - FY18-19

raw_content:
Standardized Program Evaluation
Protocol [SPEP] Report
Melbourne Center for Personal Growth
AMI Kids, Inc.
(Contract Provider)
1000 Inspiration Lane
Melbourne, Florida 32934
Primary Service: Aggression Replacement Training (ART)
SPEP Review Date(s): 02/26/19‐03/01/19
Florida Department of Juvenile Justice
Report Date(s): 02/28/19
Introduction
The Standardized Program Evaluation Protocol (SPEP) is data-driven rating system for
evaluating program effectiveness. The SPEP was developed by relying on hundreds of studies
that have examined key characteristics of programs that demonstrated their ability to reduce
recidivism. To determine a SPEP score, the program’s services are compared to characteristics
of effective programs in the research of a similar type.
The SPEP score is composed of four principle characteristics that have been shown to predict
reductions in youth recidivism; service type, quality of service, dosage (amount of service), and
youth level of risk.
 Service Type: service type points are awarded based on the group category that the
service falls within
 Quality of Service: quality points are awarded by determining the extent to which the
program is delivered in what research suggests is a high-quality manner
 Amount of Service: or “dosage” refers to the amount and duration of services that were
received
 Risk Level: refers to the level of criminogenic risk for youth involved in the intervention.
The final output of the SPEP review is the Program Optimization Percentage (POP). This rate is
derived from the Basic Score and Program Optimization Score. The POP rate is a percentage
score that indicates how the service rates in effectiveness when compared to the potential
effectiveness if optimized to match the most effective services in the research. This means the
POP rate represents how close the program’s Primary Service is to its potential for that Primary
Service Type. For example, a POP rate of 55% would indicate that the program’s Primary
Service is operating at 55% of its potential effectiveness for recidivism reduction based on the
SPEP research.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 2
Basic Score: 35 Points
1. Primary Service and Supplemental Service Types POS: 35 Points
POP: 100%
There are five Primary Service Types that have been classified into Groups with a maximum
number of points possible for rating purposes. Some Primary Service Types may also have
qualifying Supplemental Service Types that could earn a program an additional 5 points.
The Primary Service for this program is Aggression Replacement Training. The program was
awarded 30 points because the Primary Service is identified as a Group 5 Service. The specific
Sub-Component Service Type identified is Cognitive Behavioral Therapy. The Primary Service
was identified as this type of service as it is intended to correct faulty cognitions and perceptions
and provides skills individuals can use to monitor thought patterns and correct behaviors.
An additional 5 points was awarded based on a Qualifying Supplemental Service. The Qualifying
Supplemental Service was identified as None (automatic 5 points added to score), which was not
demonstrated to have been implemented.
The Primary and Supplemental Service Raw Score is equal to the sum of the Primary Service
points plus the Qualifying Supplemental Service points.
Note: Quality information is evaluated while on-site during the annual compliance review.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 3
Basic Score: 20 Points
2. Overall Quality of Service Delivery Score POS: 20 Points
POP: 100%
The Quality of Service Delivery Score is the sum of the scores for the seven treatment quality
indicators. The Program Optimization Percentage Rating determines the Overall Quality of
Service Level:
Indicator Sum Score 0-3 = Low; Sum Score 4-7 = Medium; Sum Score 8-10 = High.
Sum of all Indicator Scores (a – g below): 10 Points
Overall Quality of Service Delivery Level:
Low (Raw Score = 5)
Medium (Raw Score = 10)
High (Raw Score = 20 Points)
Basic Score: 1 Point(s)
a. Facilitator Training
Maximum Possible Score: 1 Point
All facilitator(s) of the Primary Service must have received formal training specific to the
intervention or model/protocol.
During the review period, there were 8 staff formally trained by a qualified trainer in the
Aggression Replacement Training curriculum, one Director of Case Management and seven
Outreach Specialists. Seven staff facilitated groups during the review period. Two of the Outreach
Specialists were no longer employed as of the start of the review.
Basic Score: 2 Point(s)
b. Treatment Manual/Protocol
Maximum Possible Score: 2 Points
There is a specific written manual/protocol detailing delivery of the Primary Service.
The manual provided to this reviewer was Aggression Replacement Training - A Comprehensive
Intervention for Aggressive Youth. The manual contains a script for the facilitator for the groups. It
provides the lessons and includes implementation guidelines, identifies the order the lessons
should be delivered in and includes detailed instructions for the delivery of each lesson.
Basic Score: 1 Point(s)
c. Observed Adherence to the Manual/Protocol
Maximum Possible Score: 1 Point
Upon observation of the Primary Service by the Quality Improvement reviewer, the facilitator of
that service adhered to the written protocol/manual.
The Aggression Replacement Training intervention was observed on Tuesday, 02/26/19 from
11:30-12:30. The facilitators followed the lesson according to the manual, did not read the lesson
verbatim, used their own words to convey the meaning of the discussed topic, and did so in an
easy to follow manner. The facilitators adhered to the lesson plan in every aspect.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 4
Basic Score: 2 Point(s)
d. Facilitator Turnover
Maximum Possible Score: 2 Points
Measures the extent to which facilitators of the specific intervention/service have changed as
well as gaps in service of that Primary Service.
A review of the documentation provided and an interview with the Director of Case Management
revealed there has been some facilitator turnover during the review period; however, the turnover
did not adversely affect services to the youth. The program, in the case of turnover, will utilize
other trained staff to facilitate the curriculum. There were no gaps in the service of Aggression
Replacement Training to the youth.
Basic Score: 2 Point(s)
e. Internal Fidelity Monitoring
Maximum Possible Score: 2 Points
The program has a process to monitor the delivery of the intervention to examine how closely
actual implementation matches the model protocol.
The documentation for fidelity monitoring was reviewed. The program completed fidelity
monitoring for Aggression Replacement Training for each month during the review period for each
staff member who facilitated the group. The monitoring was performed by the Director of Case
Management.
BasicScore: 1Point
f. Corrective Action based on Fidelity Monitoring
Maximum Possible Score: 1 Point
The program has a process by which corrective action is applied and demonstrated based on
the fidelity monitoring of the delinquency intervention/therapeutic service.
An interview with the Director of Case Management and a review of documentation verified the
program has a process for corrective actions based on the fidelity monitoring conducted. The
program uses internal fidelity monitoring/corrective action checklists specific to Aggression
Replacement Training. The program did not take any corrective action during the review period.
g. Evaluation of Facilitator Skill Delivering the Basic Score: 1 Point
Intervention Maximum Possible Score: 1 Point
Performance evaluations of the facilitators of the specific intervention/service include
evaluation of skill in delivering the intervention/service.
A review of all performance evaluations of each trained facilitator found entries on each which
included an evaluation of their skills in facilitating the Aggression Replacement Treatment
intervention and gave specific feedback as to their skills.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 5
3. Amount of Service ‐ Duration
Basic Score: 6 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 60%
Research indicates the target duration of 10 weeks for this type of service. Of the 55 youth in the
sample, 69% (38 of 55) reached at least the indicated target duration. Further explanation is
detailed in the Summary and Recommendations below.
Note: Dosage information (duration) is calculated from the Juvenile Justice Information System
(JJIS) Evidence-Based Services module. Duration is included for the youth in the SPEP sample.
4. Amount of Service – Contact Hours
Basic Score: 8 Points
Program Optimization Score: 10 Points
Program Optimization Percentage: 80%
Research indicates a target of 30 contact hours for this type of service. Of the 55 youth in the
sample, 80% (44 of 55) reached the indicated target contact hours. Further explanation is detailed
in the Summary and Recommendations below.
Note: Dosage information (contact hours) is calculated from the Juvenile Justice Information
System (JJIS) Evidence-Based Services module. Contact hours are included for the youth in the
SPEP sample.
5. Risk Level of Youth Served:
Basic Score: 25 Points
Program Optimization Score: 25 Points
Program Optimization Percentage: 100%
Percentage of Youth with Moderate, Moderate-High, and High-Risk Levels to Reoffend: 98%
Moderate to High Score: 12 Points
Program Optimization Score: 12 Points
Program Optimization Percentage: 100%
Table 1
Moderate = 4 youth
Moderate-High = 24 youth
High = 26 youth
Total Youth in Sample = 55 youth
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 6
Percentage of Youth with High-Risk Level to Reoffend: 47%
High Score: 13 Points
Program Optimization Score: 13 Points
Program Optimization Percentage: 100%
Table 2
High = 26 youth
Total Youth in Sample = 55 youth
The risk level score is compiled by calculating the total percent of the SPEP sample that score
Moderate to High-Risk to reoffend and also the total percent of the SPEP sample that score High-
Risk to reoffend.
Of the SPEP sample, 98% (54 of 55) youth scored Moderate to High-Risk to reoffend, for a score
of 12 points.
Of the SPEP sample, 47% (26 of 55) youth scored High-Risk to reoffend, for a score of 13 points.
Note: The latest Community Positive Achievement Change Tool (C-PACT) prior to the placement
date was used in the derivation of the risk level score. This C-PACT provides the best indication
of the risk to re-offend level of the youth when the youth was first placed in the program.
Summary and Recommendations
Program Program
Category Basic Score Optimization Optimization
Score Percentage
Primary and Supplemental Service Type 35 35 100%
Quality of Service Delivery 20 20 100%
Amount of Service: Duration 6 10 60%
Amount of Service: Contact Hours 8 10 80%
Risk Level of Youth Served 25 25 100%
Totals 94 100 94%
This SPEP report evaluates Aggression Replacement Training, an intervention delivered at
Melbourne Center for Personal Growth.
56 youth completed the program during the review period.
The program scored High for Quality of Service Delivery.
The targeted number of weeks for this specific intervention is 10 weeks; the targeted number of
hours for this specific intervention is 30 hours. After additional review and discussion with the
Peabody Research Institute at Vanderbilt University, the Department determined it was within
SPEP standards to deviate from the standard dosage targets for Group 5 services (15 weeks; 45
hours).
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 7
The program earned 6 points for Amount of Service: Duration. Of the 55 total youth sampled, only
46 included dosage with end dates in the EBS Module. Of those youth with correct dosage, 38
received at least the recommended weeks of service. Youth in the sample completed between 8
and 11 weeks of service, with an average of 10 weeks.
The program earned 8 points for Amount of Service: Contact Hours. Of the 55 total youth
sampled, all 55 included dosage in the EBS Module. Of those youth, 44 received at least the
recommended hours of service. Youth in the sample completed between 23 and 30 hours of
service, with an average of 29 hours.
The program was awarded 25 available points for Risk Level of Youth Served. This is calculated
using data from the Community - Positive Achievement Change Tool (C-PACT) assessment. This
score reflects youths' most recent C-PACT score prior to placement at the program. The program
itself has no control over youths' C-PACT risk level because the scored assessment was
administered prior to the youths' admission.
RECOMMENDATION(S):
Melbourne Center for Personal Growth can maintain their SPEP Quality of Service Delivery score
by continuing the practices in place at the time of this review.
Melbourne Center for Personal Growth can optimize their SPEP Amount of Service score by
ensuring that dosage for all youth is recorded accurately in EBS and by ensuring that youth
receive the full targeted dosage of service.
Department of Juvenile Justice | Standardized Program Evaluation Protocol [SPEP] Report 8