21CCLC Statewide Evaluation Plan 2023

Nita M. Lowey 21

Century Community

Learning Centers Statewide Evaluation Plan

September 22, 2023

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 2 of 14

Purpose

Evaluation is a systematic assessment of the design, implementation, and/or results of an

initiative for the purposes of program improvement

. As such, this evaluation plan will be used

to guide activity within the Nita M. Lowey 21

Century Community Learning Centers Program

(21C) starting in July of 2023 and can help inform other initiatives that support out-of-school

time (OST) learning. Connected strategies may include but are not limited to the work of

Agency of Education’s (AOE) supportive accountability activities and associated technical

assistance, grant administration and monitoring; and it can act as a road map to program

quality and coherence to support the State and local grantees achieve desired outcomes for

Vermont learners. AOE will also track its aligned work and strategies to these goals and results.

Introduction and Background

This evaluation plan aligns with the strategic goal adopted by the Vermont State Board of

Education to “Ensure that Vermont’s public education system operates within the framework of

high expectations for every learner and ensure that there is equity in opportunity for all.” In

addition, the federal 21

CCLC initiative over time has been designed to support Vermont’s

vision for Vermont learners articulated in Title 16, the Education Quality Standards, and the

Vermont Portrait of a Graduate. Most recently, this plan also supports the pillars of Vermont’s

Education Recovery and Revitalization Framework: Academic Achievement, Social-Emotional

Learning and Well-Being, and Educator Well-Being.

Vermont’s 21

CCLC program and this evaluation plan are committed to acknowledging

connections to other state and federal programs, funding sources and legislation, such as

Vermont’s Act 77 (2013), Act 173 (2018), Act 1 (2019), Act 28 (2021), Act 67 (2021), and others,

and federal programs including those in the areas of nutrition, federal title programs and

special education to name just a few. The AOE recognizes the importance of braiding and

blending, where possible, funding across systems and aligning and integrating programs,

reporting and evaluation systems, and performance and quality indicators.

This plan updates and replaces the last plan finalized in 2014 that informed Vermont’s 21CCLC

program through the pandemic of 2020. Due to the results of the evaluation conducted during

the pandemic, the focus of this plan is on implementation, with a goal to meet and/or exceed

pre-pandemic performance.

As such, the updated evaluation plan includes, overall, 13 new

items, 20 revised items and 14 items were retained. The following codes are used in the chart

below to indicate ***=new, **=maintained, or *=revised.

We are using the same definition being used throughout the 21

CCLC system in concert with USDOE.

Creation of this plan took place over two years and involved meaningful feedback from over [35] people including many local,

state, and national level 21

CCLC stakeholders. Emanuel Betz was the author.

What

New # compared to old plan

Maintained#

Revised #

Goals

Results

Indicators

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 3 of 14

The following principles guided the design of this updated plan:

Principle 1: Design to avoid replicating existing scholarly research and to complement but not

duplicate State and Agency goals and Federal Reporting “GPRA” indicators.

Principle 2: Focus on a unified statewide evaluation, where both state and local programs will

use and/or adopt it.

Principle 3: Support accountability, implementation, and continuous improvement efforts.

Principle 4: Identify projects that exceed the result statements in a majority of items as

statewide models that can offer new approaches to support growth and improvement.

Principle 5: Maintain continuity with historic data indicators while engaging in improvement

activities.

This plan will be in effect for at least five years or until July 1, 2028. In the first year of the plan,

and in subsequent years for new projects as they are funded, local grantees will be asked to

voluntarily adopt all or any parts of the plan to complement and/or extend their local

comprehensive evaluations. In addition, in the first year of the plan, indicators and results may

be amended at the discretion of the Agency of Education. Using this plan as a broad-based

growth model over time, the Agency seeks to have all funded programs meeting or exceeding

the results statements described below.

Goals, Results, and Indicators

Goal 1: 21C funded programs assure access and equity for all children, youth, and their

families.

Result 1.1 21C funded programs serve children and youth who have fewer opportunities to

learn

outside of the school day. **

Result 1.2 21C funded programs are open for enough hours, days, and weeks to meet

children’s, youth and family needs during the summer and school year. **

Result 1.3 21C funded programs have a varied base of attendees within the federal hour

bands.

***

Learning is meant to be an inclusive phrase, for example, validating all areas of multiple intelligence theory.

The three research bands include: 90-179 hours, 180-269 hours, and 270 hours plus.

CCLC State Evaluation Plan (Revised: Page 4 of 14

September 22, 2023)

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds the

Result

Statement

Why this matters

1.1a.21C funded sites serve

economically disadvantaged

participants [ free and reduced lunch

eligible] at rates that meet or exceeds

the school rate.**

Under the school

average by minus

10% or more

TBD 2023

parity

= > 10%

Research shows that

low income/ poverty

includes many who

have experienced

opportunity gaps

and lower

educational

attainment

1.1b.21C-funded sites serve youth

with disabilities [IEP/504} at rates

that meets or exceeds the school

rate.**

Under the school

average by minus

20% or more

TBD 2023

parity

= > 15%

While just one

indicator, tracking

this metric can bring

focus to serving and

caring about

potentially

marginalized

children and youth.

1.1 c 75% of 21C funded sites are in

communities where there are no

other comprehensive (>150 day)

afterschool options *

<50%

75%

>75%

Like 1.1a, Funds will

go to low income

and “low

opportunity”

communities at the

state level.

1.2 75% of 21C-funded sites meet or

exceed the state average range and

are open for enough days and hours

to meet student and family needs

during the school year and summer

***

<10%

Average

= > 10%

Sustained program

duration is tied to

positive outcomes in

the research

literature.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 5 of 14

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds the

Result

Statement

Why this matters

1.3a 75% 21C funded sites have

youth attending

in at least 5 of 6

federal hours bands *

75%+

This indicator

reflects and

measures how

programs are

supporting the

diverse needs,

interests, and

program access for

afterschool

participants.

1.3b Site, Project, and Statewide

Attendance rates exceed 50% for the

research-based hour bands

***

<50%

50%

>50%

This replaces the

previous federal

collection

methodology and

previous state

regular attendee

designation of 30

days or 60 hours.

A minimum of three must be attained to count in an hour band.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 6 of 14

Goal 2: All 21C Funded programs are of high quality

Result 2.1 21C funded programs provide diverse

opportunities for learning, including social

emotional growth.***

Result 2.2 21C funded programs encourage participants to develop skills by experiencing

interests and/or content in depth. **

Result 2.3 21C funded programs use data for continuous improvement. *

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds the

Result

Statement

Why this

matters

2.1a Larger sites with more than 100

(not counting PreK) youth offer a

minimum range of 15-25 unique

program choices/offerings

within

any given week or session; or a 10-

15 minimum range for smaller

schools of under 100. *

<15

<10

Within the

range

= >20% of avg

Numerous

authentic choices,

engaging options,

keeps

programming

continually rich,

flexible, and

exciting.

2.2a Each 21C-funded site’s

programs will result in a minimum

of 5 concrete examples of

culminating end products or

performances

during the school

year***

Programs with

identified ends

and/or a clear

purpose combined

with managing

effective and

developmentally

appropriate

pedagogy over

time is the

foundation for

quality.

Diverse is meant to convey the need for the idea of a multiplicity of program types (e.g., different content) as well as building to

multiple approaches or initiatives (e.g., enrichment, tutoring, youth leadership) so that a program is robust and multi-dimensional

in its implementation over time.

“Unique program offerings” are listed programs in a timebound program brochure yet may include other intentionally designed

regular program components that occur within the same time period.

There are many types of examples ranging across different content areas and experiences from short projects to evidence of

accumulated knowledge or skills gained to specific artifacts or evidence created across time.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 7 of 14

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds the

Result

Statement

Why this

matters

2.2b Each 21C funded site serving

grades (4-12) will offer at least 1

program offering

(3 for sites in

schools with more than 100 youth)

that meets for at least 30 hours and

systematically builds lasting

physical artistic, academic, or other

skill sets.

TBD Below

the average

TBD average

TBD Above

the average

In addition to the

above reasons,

these programs

can support

outcomes and

growth at a deeper

more personal, or

transformative

level.

See: VT Portrait of

A Graduate

2.3a 90% of 21C projects complete a

continuous improvement

recognized process annually such as

PQA or SELPQA after year 1 of

funding. ***

Partial

process

completion

Observation,

meetings,

scores, and

plan

completion

Evidence of

executing plan

improvements

Every high-quality

program should

use a validated

continuous

improvement

methodology as

part of its core

system

infrastructure.

2.3b 90% 21C projects beyond year 3

report on local evaluation

measurements beyond surveys and

state requirements.*

Metrics

exist but

may not be

active

and/or

SMART

At least 3

metrics exist

with

outcomes

reported

More than 5

reported

SMART

outputs and/

or outcomes

Projects are

expected to use

meaningful local

measures that

complement or

extend state and

federal reporting

and evaluation

systems.

In this case a program offering is meant to be a specific long-term choice with substantial practice such as Jazz Club, Girls on the

Run, or a musical production for example.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 8 of 14

Goal 3: All 21C funded programs are led by effective leaders

Result 3.1 21C funded programs are led by knowledgeable and skilled leaders.***

Result 3.2 21C funded programs utilize high quality staff to run programs.**

Result 3.3 21C funded programs have appropriate staff retention rates.**

Result 3.4 21C funded staff are valued for their work.*

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds

the Result

Statement

Why this matters

3.1 a: 95% of 21C programs will be

led by directors with significant

levels of knowledge and skill ***

(B.A or higher)

3.1 b: 75% 21C sites will be led by

site coordinators with significant

levels of knowledge and skill***

(B.A or higher)

<75%

equal

Average

greater

75%+

21C funded

programs do not

require a degree to

work in programs

and as such values

every worker

regardless of

degree or

credential attained.

Tracked data

includes from high

school completion

through to a

doctoral degree

attainment.

3.2 21C sites will be staffed by at

least 33% licensed educators and/or

equivalent subject matter

experts.***

ii11

<10%

Average or

33%,

whichever

is lower

>10% avg

Leveraging

talented + skilled

staff from both the

school and

community

enhances the

diversity and

breadth of

programming and

associated

relationships built.

Subject matter experts are proven educators from the school or community who have mastery of a content area and/or may be

working toward a credential of value.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 9 of 14

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds

the Result

Statement

Why this matters

3.3a. After 2025 75% of project

directors have been in their position

for at least 3 years. ***

<75%

75%

>75%

Leadership

continuity is

essential for

consistency and

growth.

3.3b. 75% of site coordinators have

been in their position for more than

one year or meeting the state

average, if greater than one year.***

<75%

75%

>75%

Leadership

continuity at this

level is essential

for consistency and

growth in a

position that is

often a stepping

stone for

employees.

3.4 80% of projects implement at

least one strategy intentionally

designed to value their staff’s

work.*

<80%

80%

>80%

Implementing

authentic

strategies in this

area can be

intrinsically

motivating and

support building a

positive

organizational

culture.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 10 of 14

Goal 4: All 21C funded programs develop youth leadership, agency, and voice

Result 4.1 21C funded program sites implement intentional strategies that support youth

collaboration

Result 4.2 21C funded projects offer explicit opportunities for youth to build leadership

experience.

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds

the Result

Statement

Why this

matters

4.1 100% of 21C funded program

sites use multiple strategies that

support youth collaboration.

<100%

100%

Youth

collaboration is an

inclusive and

flexible tenet that

validates multiple

approaches in

tracking Goal 4.

4.2a 50% of 21C funded projects

in secondary programs have youth

who teach or lead programs*

40%

50%

>60%

This is an

emergent

approach that fits

well into the

afterschool

context. Programs

should only

attempt this

approach when

the system

supports are

ready.

4.2b 25% of Projects after year 10

have a documented youth

leadership program or system in

place.

<25%

25%

>25%

Programs that

achieve this likely

have a youth-led

culture in place.

Youth collaboration relates to an intentional action a project takes to include all children, youth, and students in the development

and implementation of quality programming and activity in a developmentally appropriate manner. As such, it is broadly inclusive

of “voice and choice” type activity as well as more advanced examples that could include: Informal querying, listening and acting

on recommendations, co-leading activities or holding jobs/tasks as helpers or buddies, teaching as lead/with adult

assistance/mentorship, apprentice or counselor in training type programs, Capstone integrations, youth led performances, student

council work, part of district hiring team for directors, advocacy activity of varying types, acting as youth ambassadors: state,

speaking at school board meetings or presenting to town or other officials [to name some actual examples]

ibid

A leadership program system would have documented program components/processes that are active such as in the areas of:

clear expectations, on-boarding, training, intentional support structures, evaluative activity.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 11 of 14

Goal 5 All 21C funded programs are sustainable

Result 5.1 21C funded programs are integrated with school day systems.***

Result 5.2 21C funded programs utilize diverse sources of funding.**

Result 5.3 21C funded programs include meaningful community partnerships.**

Result 5.4. 21C funded programs are cost-effective.***

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds

the Result

Statement

Why this

matters/notes

5.1 75% of project directors are

active members of the district

leadership or another district level

administrative team ***

<75%

75%

>75%

Evidence of deep

school-community

integration leads to

quality + sustainable

outcomes.

5.2 90% 21C-funded projects that

have been in operation for more

than five years have at least five

different sources of funding

contributing more than 50% to

their annual operating budget ***

<90%

90%

>90%

This goal was used

and met in the first 20

years of the program.

(This may need to be

adapted after 2023

actuals are calculated)

5.3a 100% 21C-funded projects

work with a minimum of 5

community partners

that

contribute at least $100 to the

project annually. ***

5.3b 100% 21C-funded projects

work with at least one significant

partner that contributes a

minimum of $1000 in resources to

the project annually. ***

90%

<100%

95%

100%

Both minor and major

partners and

collaborators are

essential contributors

to a high- quality

sustainable

community learning

center.

Note that language

includes “at least”

and” at a minimum,”

much more can be

achieved excluding

important non-

financial supports.

A partner is an organization that shares ownership of the project and its intended results. Partners may assume responsibility,

manage, contribute, or create activity that supports the project. For communities without organizations in reasonable geographic

proximity or of sufficient quality, an individual or organized group may also act as a partner.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 12 of 14

Indicator

Below the

Result

statement

Meets the

Result

Statement

Exceeds

the Result

Statement

Why this

matters/notes

5.4a per total hour costs are not

over 30% of average when

comparing similar projects by type

and size ***

5.4 b) per attendee costs are not

over 30% of average when

comparing similar projects by type

and size ***

Within

average

range

Within

average

range

These measurements

will be much more

precise than the 2014

plan to understand

what a sustainable

high-quality program

should cost.

CCLC State Evaluation Plan (Revised:

September 22, 2023)

Page 13 of 14

Every three years-GPRA Aligned Analysis

Indicator

Below the

expected

Result

Meets the

expected

Result

Exceeds the expected result

In-school attendance rates for

elementary and middle/high school

youth increase by at least three days for

youth attending afterschool regularly

when compared with low hour

participation**

(GPRA #3)

3-6

Appendices

The 21C Federal level hour bands:

(21C Legislation Related to Evaluation)

State Level:

(14) describes how the State educational agency will evaluate the effectiveness of programs and

activities carried out under this part, which shall include, at a minimum—

(A) a description of the performance indicators and performance measures that will be used to

evaluate programs and activities with emphasis on alignment with the regular academic

program of the school and the academic needs of participating students, including performance

indicators and measures that— (i) are able to track student success and improvement over time;

(ii) include State assessment results and other indicators of student success and improvement,

such as improved attendance during the school day, better classroom grades, regular (or

consistent) program attendance, and on-time advancement to the next grade level; and (iii) for

high school students, may include indicators such as career competencies, successful

completion of internships or apprenticeships, or work-based learning opportunities;

(B) a description of how data collected for the purposes of subparagraph (A) will be collected;

and

part

(Grantee Level)

(2) PERIODIC EVALUATION.— (A) IN GENERAL.—The program or activity shall undergo a

periodic evaluation in conjunction with the State educational agency’s overall evaluation plan

as described in section 4203(a)(14), to assess the program’s progress toward achieving the goal

of providing high-quality opportunities for academic enrichment and overall student success.

(B) USE OF RESULTS.—The results of evaluations under subparagraph (A) shall be— (i) used

to refine, improve, and strengthen the program or activity, and to refine the performance

measures; (ii) made available to the public upon request, with public notice of such availability

provided; and (iii) used by the State to determine whether a subgrant is eligible to be renewed

under section 4204(j).

CCLC State Evaluation Plan (Revised: Page 14 of 14

September 22, 2023)