University of Cincinnati
2610 McMicken Circle, Teachers-Dyer Complex, Suite 150
Cincinnati, Ohio, 45221
http://www.uc.edu/evaluationservices
Columbus State Community College
Strengthening Mobile Application
Resources and Technical Training
(SMARTT) Project:
Year 1 Evaluation Report
Prepared for:
Jon Lundquist, M.B.A., Professor, and Interactive Media
Coordinator
Columbus State Community College
June 2018
Columbus State Community College
Strengthening Mobile Application Resources
and Technical Training (SMARTT) Project:
Year 1 Evaluation Report
Evaluation Services Center
Jacinda K. Dariotis, Ph.D., M.A.S., M.A., M.S., Director & Professor
Report prepared by:
Catherine V. Maltbie, Ed.D., Research Associate
Edie Morris, Ph.D., Research Associate
Audra Morrison, M.A., Assistant Director
Amy D. Short, M.H.S.A, Associate Director
Rachel Smith, M.H.Sc., Program Manager
© 2018 Evaluation Services Center
University of Cincinnati
PO Box 210175
Cincinnati, OH 45221-0175
Tel: (513) 556-3900
Fax: (513) 556-3516
http://www.uc.edu/evaluationservices/
E-mail: [email protected]du
Suggested citation:
Maltbie, C. V., Morris, E., Morrison, A. B., Short, A. D. Smith, R.R., & Dariotis, J.K., (2018,
June). Columbus State Community College Strengthening Mobile Application Resources and
Technical Training (SMARTT) Project: Year 1 Evaluation Report. Cincinnati, Ohio:
Evaluation Services Center, University of Cincinnati.
i
Columbus State Community College Strengthening Mobile Application
Resources and Technical Training (SMARTT)
Year 1 Evaluation Report
Table of Contents
Purpose of Evaluation ........................................................................................................ 1
Evaluation Questions ......................................................................................................... 2
Evaluation Procedures ........................................................................................................ 3
Participants ............................................................................................................................................................ 4
Instruments ........................................................................................................................................................... 4
Data Collection Procedures ............................................................................................................................... 4
Data Analysis Strategies ...................................................................................................................................... 4
Results................................................................................................................................. 5
Revisions to Evaluation Design and Plan ........................................................................................................ 5
Establishment of Columbus State Community College Mobile Application Pathways ........................... 6
Recruitment into SMARTT Pathways .............................................................................................................. 7
Conclusions ........................................................................................................................ 8
Recommendations .............................................................................................................. 9
Appendices ....................................................................................................................... 10
List of Tables
Table 1. Courses to be Revised for SMARTT Curriculum Implementation ..................... 5
List of Appendices
Appendix A. Business and Industry Leadership Focus Group Protocol ......................... 11
Appendix B. Updated Evaluation Design ........................................................................ 13
SMARTT Project Year 1 Evaluation Report 1
Columbus State Community College
Strengthening Mobile Application Resources
and Technical Training (SMARTT):
Year 1 Evaluation Report
Purpose of Evaluation
The University of Cincinnati Evaluation Services Center (UCESC) has been contracted
by Columbus State Community College (CSCC) to conduct an evaluation of the three-year
Strengthening Mobile Application Resources and Technical Training (SMARTT) Project,
funded by National Science Foundation (NSF) Advanced Technological Education (ATE)
award #1700519. The project’s main goal of developing an education program for Mobile
Application Development Technicians at Columbus State Community College is to meet the
increased workforce demand by Ohio regional industry and to contribute to the diversity of this
region’s workforce. Companies are changing the way they think and altering hiring patterns
from traditional four-year degrees to graduates with certificates and associate degrees for their
open positions, valuing technical knowledge over a liberal arts degree. The SMARTT project
intends to address the shortage of mobile developers through the creation of two associate
degree programs, Mobile App Design in Interactive Media and Mobile App Development in
Computer Science, and a collaborative certificate at Columbus State Community College.
The goal of the SMARTT project is to develop an education program for mobile
application development technicians at Columbus State Community College to meet the
increased demand by Ohio regional industry and to contribute to the diversity of the region’s
work force. The project objectives from the proposal include:
1. Establish a Mobile Application Development Technology 2+2+2 pathways featuring:
a. A model articulation agreement from high school to community college and then
to university level baccalaureate programs.
b. The expansion of the existing Industry Advisory Committee into a true BILT under
the guidance of Dr. Ann Beheler from the National Convergence Technology
Center (CTC) so industry co-leads work and provide maximum benefit for students.
c. A collaborative Mobile Design and Development certificate curriculum led by the
BILT.
2. Create a collaborative state-of-the-art curriculum leveraging existing coursework in
Interactive Media and Computer Science:
SMARTT Project Year 1 Evaluation Report 2
a. An online curriculum featuring digitized, adaptive learning object methodology.
b. A project-based learning capstone supported by a scalable, state-of-the-art device
lab that that will provide the real-world work experience in designing and using
mobile application development technology-related software and hardware.
3. Recruit underserved, disadvantaged populations (defined as low-income, first generation
students) to the new SMARTT (Mobile Application Design and Mobile Application
Development) pathways through partnerships with high school districts, special interest
programs, and non-profit organizations.
The purpose of the three-year evaluation is to provide formative and summative
evaluation data regarding the progress made toward these goals and outcomes of the project.
For Year 1, the project team recruited a Business and Industry Leadership Team (BILT).
Evaluation activities focused on updating the evaluation plan to reflect project activities and
reviewing project documentation related to curriculum development and partner involvement in
this process (e.g., BILT members). During Years 2 and 3, the evaluation will focus on
curriculum design and planned curricular and course revisions using faculty and student input.
Evaluation Questions
The evaluation team, in collaboration with the project team, revised the evaluation
questions, along with the evaluation design and plan. The updated guiding questions are
outlined below.
Formative Questions (FQ)
1. What activities, processes, and practices contributed to the establishment of the
College’s Mobile Application Development Technology 2+2+2 pathway?
2. How did the project facilitate leveraging the existing coursework to create a state-of-the-
art curriculum leading to an education program for mobile application development
technicians?
a. Development of a collaborative, interdisciplinary capstone course focused on
project-based experiential learning.
b. Integration of project-based learning and soft skills development in all courses.
c. Approaches to faculty development strategy.
d. Delivery of blended curriculum.
e. Development and use of the SMARTT project device lab.
SMARTT Project Year 1 Evaluation Report 3
3. To what extent did the project recruit disadvantaged populations to the new SMARTT
pathway?
Summative Questions (SQ)
1. To what extent has the project established a Mobile Application Development
Technology career education 2+2+2 pathway for a disadvantaged/underserved student
population?
a. Establishing and executing model articulation agreements with high schools and
university level baccalaureate programs.
b. Developing and expanding curricula with industry input from the BILT, such as
the development of a new Collaborative Interdisciplinary Mobile Design and
Development (CIMDD) certificate.
c. The number of underserved and female students who:
i. Enroll in the new Mobile certificate
ii. Complete the new Mobile certificate
iii. Enroll in the Mobile AAS degree program
iv. Remain in Mobile AAS degree program
v. Complete capstone projects
vi. Enter the workforce after completing their certificate or capstone
project
vii. Complete the Mobile AAS degree program
viii. Begin matriculation to a four-year institution.
2. To what extent are students in the Mobile Application Development Technology career
education 2+2+2 pathways satisfied with the program?
Evaluation Procedures
Evaluation procedures for Year 1 of this project were primarily formative and reflected
the fact that the project team was focused on creating the Business and Industry Leadership
Team (BILT) to provide guidance for the planned curriculum development activities. The
evaluation and project teams met in person in December 2017 and held virtual meetings on a
regular basis to discuss progress made and specific project activities. Year 1 external evaluation
activities are outlined below.
SMARTT Project Year 1 Evaluation Report 4
Revised project evaluation plan.
Obtained human subjects approval from UC Institutional Research Board (IRB).
Reviewed BILT involvement in the development of the SMARTT curriculum.
Participants
The SMARTT study will be recruiting participants in phases during the three-year grant
period. In Phase 1, during the first year, the project team recruited BILT members who will
provide advising and leadership into industry needs in relation to application design and
development. The members of the BILT are listed in the SMARTT NSF Annual Report. At
this time, these are the only participants included in this study.
In Fall 2018, BILT members participate in a focus group discussion conducted by the
evaluation team. Future study phases will include faculty who helped re-design the curriculum
and undergraduate college students who are currently enrolled in the existing curriculum and
future students who enroll in the Mobile Application pathways courses. University of Cincinnati
IRB protocols will be amended as the project progresses.
Instruments
A BILT focus group protocol has been developed in Year 1. The protocol includes
eight open-ended questions to gain feedback on the project’s progress and their involvement in
the curriculum development process. The protocol is shown in Appendix A of this report and
has been approved by the University of Cincinnati’s Institutional Review Board.
Data Collection Procedures
The evaluation team obtained curriculum development materials from the project team.
These documents were tracked and reviewed to identify evidence of collaboration with the
BILT and completion of curricular materials.
Data Analysis Strategies
The evaluation team reviewed project team documents during this Year 1 that included
PowerPoint presentations presented at WebEx meetings among the project team and BILT
members, voting documents from BILT meetings, curriculum development documents for
various courses (IMM Design and CSCI), as well as the plan of studies, and the draft of the
annual report for NSF.
SMARTT Project Year 1 Evaluation Report 5
Results
At this time, the evaluation will address questions 1 and 2 (formative) since student
recruitment for the pathways is scheduled to begin following approval of the developed
curriculum by the state. The project team used Year 1 to gather details about knowledge, skills,
and abilities needed by employers in the mobile application field with involvement from the
BILT. They used this input to revise existing courses and to create new courses that are part of
the CSCC Mobile Application Development Technology 2+2+2 pathways in mobile application
development and in mobile application design. This curricular approval process has taken
longer than initially expected; however, this increased time has allowed the evaluation team, in
collaboration with the project team, to revise the evaluation plan to reflect these intended
outcomes and add rigor, including follow-ups with students and a comparison group. The
evaluation plan now includes data collection activities among a group of undergraduate students
who will be taking the current CSCC foundational courses in academic year 2018-19 and will be
a comparison group for SMARTT students who begin in Fall 2019 to take the revised CSCC
courses (for the 2019-20 academic year).
Revisions to Evaluation Design and Plan
Course approval timing (planned Fall 2019 implementation of SMARTT curriculum)
has allowed the evaluation team, in collaboration with the project team, to change the
evaluation design to establish a comparison student group for the 2018-2019 academic year,
increasing the study’s rigor. During the 2018-2019 academic year, Columbus State Community
College students will be able to enroll in five identified courses, that are part of the current
curriculum and are bring revised for the SMARTT curriculum implementation in Fall 2019.
Table 1. Courses to be Revised for SMARTT Curriculum Implementation
Course Number Course Title
CSCI 1650 Programming Fundamentals for iOS
CSCI 1660 Programming Fundamentals for Android
CSCI 2650 iOS Mobile Apps Development
CSCI 2660 Android Mobile Apps Development
IMM 2372 Adobe Phonegap
Student course evaluations will be collected and analyzed for these courses in Years 2
and 3 in order to compare results between the courses before implementation and following
implementation. Additional surveys to determine faculty members’ and students’ perspectives
about the current mobile program are being developed and will be administered to students in
SMARTT Project Year 1 Evaluation Report 6
these courses as well as the faculty teaching these courses. The student survey will include
questions assessing student self-efficacy and confidence to succeed in mobile applications
careers, employability and employment options, and an evaluation of the program. A rubric to
assess student performance in projects will be developed and piloted during the 2018-19
academic year. The project and evaluation teams have no control over when comparison
students started in their program; thus, the evaluation team will administer a post-survey only
when students graduate, either at the end of Fall 2018 or Spring 2019 semesters.
Establishment of Columbus State Community College Mobile
Application Pathways
The project team sought feedback from the BILT regarding the knowledge and skills
needed by employees in the field while developing the AAS degree programs in Mobile
Development and Mobile Design as well as the related certificate. On December 14, 2017, the
Columbus State faculty met with the BILT to discuss program and certificate development. Dr.
Ann Beheler, Co-PI/Executive Director of Collin College, National Convergence Technology
Center and the evaluation team (Jacinda Dariotis, Catherine Maltbie, and Edith Morris) from
Evaluation Services Center at the University of Cincinnati were also in attendance. The drafted
curriculums were shared with the BILT during a WebEx meeting on March 20, 2018. All
project team and evaluation team members participated in WebEx meeting.
At the December 2017 in-person meeting, all project team members, Computer Science
and Media Design faculty, and evaluation team members were introduced to the BILT
members. Ann Beheler, Co-PI led a discussion where BILT members identified and rank
ordered pertinent knowledge, skills, and abilities needed for new hires and current employees
working in the mobile application development and design fields. Specifically, BILT members
were asked to identify what makes employees in the mobile application field successful. An
overview of the conclusions reached in this meeting included: 1) knowledge of more than one
program/system was important, as was good communication skills and the ability to spring
back from failure; 2) many positions are contract positions, and hence there was concern about
getting and keeping qualified personnel; 3) demonstrating professional growth as a new hire was
important, and an online portfolio can show both performance and growth and should be
brought to job interviews.
At the March 20, 2018 WebEx meeting, the project team reviewed the program of study
drafts created with the information gathered in the December 2017 meeting. Explicit
connections between BILT comments and the plan of study were highlighted. As an overview,
three changes were made to the proposed curriculum based on the input of the BILT members.
Specific changes included: 1) removing system programming fundamentals and adding a course
on prototyping to the mobile app design program; 2) a web-based development course was
removed and replaced with Javascript, which is more useful in a hybrid world; and 3) Mobile
Analytics became a required course rather than an elective.
SMARTT Project Year 1 Evaluation Report 7
Within the above curriculum, the project team revised current courses and developed
seven new courses. These activities included the development of a collaborative,
interdisciplinary capstone course focused on project-based experiential learning as well as the
integration of project-based learning and soft skills development (such as collaborative,
negotiating, planning, and organizational skills) into all courses. The proposed curriculum will
be delivered in a blended manner, combining hands-on experiences in a newly developed device
laboratory environment with online learning. The project team is currently developing an
equipment list for the SMARTT project device lab. This new curriculum and courses are
currently in final approval stages by the internal Columbus State Community College
Curriculum Board, followed by the Ohio Department of Higher Education, and the Higher
Learning Commission and students should begin enrollment Fall 2019.
Regarding progress made toward the faculty development strategy, four members of the
project team registered to attend the Working Connections IT Faculty Development Institute in
July 2018. The faculty attending these workshops hope to incorporate what they learn into the
mobile courses and the project team hopes to learn how to host a mobile-focused track in the
future.
A follow-up meeting with BILT members is planned for Fall 2018. The evaluation team
plans to conduct a focus group with the BILT members to assess their level of satisfaction with
the input process used during curriculum development and to gauge whether or not they felt
that their input was considered when decisions were made. The project team will solicit
recommendations for equipment to be included as part of the device laboratory.
Recruitment into SMARTT Pathways
Student recruitment activities did not occur in Year 1. The project team will create
materials that will be used to recruit students during outreach events in high schools and at
recruitment events at Columbus State Community College after the curriculum is approved,
hopefully in Winter 2019. The project team continues to work to guide the SMARTT project
curriculum through the approval process at the college and state levels. The 2+2+2 pathways
will be supported as the project team creates model articulation agreements with high school
and four-year university partners to create pathways for students from high school to two-year
colleges, and then to four-year colleges.
SMARTT Project Year 1 Evaluation Report 8
Conclusions
The SMARTT project is well underway in terms of its main project goals: creating new
pathways to provide options for high school students from high school through two-year
colleges into four-year colleges, and to develop a regional workforce in mobile applications.
Year 1 activities consisted principally of developing a business and industry leadership team to
help guide the curriculum development process, and shepherding the revised curriculum
through the approval process at college and state levels. These activities are summarized below.
Since the revised curriculum is not being taught until Fall 2019, a comparison group,
which contributes more rigor to the evaluation, is planned for the 2018-2019 academic
year. This will allow the evaluation team to collect student data pertaining to course
evaluations and student products generated in five courses that are currently being
taught at Columbus State Community College that are being modified as part of the
SMARTT Mobile Application curriculum.
A curriculum was successfully developed. Curriculum outlines and course syllabi were
submitted for approval and faculty are beginning to plan lessons and activities. The
project team is planning to outfit the device lab and to attend the 2018 IT Institute for
ideas that can be adapted or adopted into the SMARTT pathways.
Documentation supports the conclusion that input from the business and industry
leaders was considered and used to develop the curriculum. After the first meeting, the
BILT team ranked knowledge, skills and abilities, and related these directly to the plan
of study. The second round of feedback refined and revised the courses and the
curriculum. The level of satisfaction from BILT members with the input process used
during curriculum development and ideas for efficiently obtaining feedback in the
future will be obtained during a Fall 2018 focus group.
SMARTT Project Year 1 Evaluation Report 9
Recommendations
In Year 2, the SMARTT project will move into implementation, establishing a comparison
group and finalizing curriculum revision and approvals.
• The project team and evaluation team will continue to work closely to ensure student
assessment and evaluation measures are appropriate for learning objectives. The data
collection process needs to be efficient for both students and faculty.
• The project team will continue regular meetings with the BILT to obtain feedback and
recommendations as lessons and activities are finalized for the revised and added
courses. The project team is focused on the device laboratory for immediate needs, but
all courses’ student learning objectives will need to be assessed for student achievement
levels and for continuous improvement of the programs.
• The project team should monitor recruitment outcomes to determine if additional
efforts need to be made to attract high school and other students into the SMARTT
degree and certificate programs. Recruitment should begin with students currently at
Columbus State Community College since the fundamental courses are currently taught
and will not change as the new degrees are implemented.
• The relationship between the certificate and degree programs needs to be further
delineated so that it is easily understood what is needed for each and a path to
completion is outlined and understood. Plans of study for each pathway will help in this
regard.
SMARTT Project Year 1 Evaluation Report 10
Appendices
SMARTT Project Year 1 Evaluation Report 11
Appendix A. Business and Industry Leadership
Focus Group Protocol
SMARTT Project Year 1 Evaluation Report 12
Open-Ended Discussion Guide
SMARTT-Phase 1 BILT Focus Group
Introduction
Hello, I am [XXXX] and this is [YYYY]. We are from the University of Cincinnati Evaluation Services
Center. We would like to ask you about your experiences with the SMARTT Project (Strengthening
Mobile Application Resources and Technician Training).
The information you provide will help improve the program. This information sheet describes the
purpose of this study and is for you to keep. [REVIEW THE INFORMATION SHEET. MAJOR
POINTS TO HIGHLIGHT: Your participation is voluntary. Your name will not be included with any of
the findings; please be as honest and candid as possible. To respect other participants, we ask that
you do not share what is discussed in this group with anyone else. We will audio record the session
to ensure we are accurately capturing the ideas. No one outside the evaluation team will hear these
recordings.] Please remember to talk one at a time and speak up as much as possible so that we
can hear what you have to say. Do you have any questions before we begin?
1. We would like to obtain some background information. What is your affiliation with the
SMARTT project?
a. What is your position?
b. Please describe company or organization.
c. What expertise do you bring to the project?
d. How many SMARTT events have you participated in?
i. What were they?
e. How many CS or mobile design students have you hired in the last year? From
CSCC?
2. If you attended SMARTT events, do you provide input?
a. Did you feel it was valued? Why or Why not?
b. How were your comments addressed?
3. How has your input been incorporated into the SMARTT curriculum?
a. Directly or indirectly?
b. What is your reaction to the current progress made for curricular development?
4. Do you plan to continue participating in SMARTT events?
a. In what capacity?
b. Why or why not?
5. In what ways does the SMARTT program support the educational development of new hires
in your field?
a. In general terms, please describe what you expect from a graduate of the new
mobile technology program at CSCC?
i. Were your expectations different, or the same, as others you have hired in
the past?
1. If yes, what was different?
6. In what ways has the SMARTT project developed a pathway for high school to AAS to BS
for regional students (2+2+2)?
7. What is working well related to the SMARTT project? What needs improvement?
8. Is there anything we have not discussed about the program that you would like us to know?
9. Other comments?
SMARTT Project Year 1 Evaluation Report 13
Appendix B. Updated Evaluation Design
SMARTT Project Year 1 Evaluation Report 14
Strengthening Mobile Application Resources and Technician Training
(SMARTT) Project
Columbus State College
Evaluation Design and Plan – Updated June 2018
Project Goals and Objectives
The goal of this project is to develop an education program for mobile application development
technicians at Columbus State Community College to meet the increased demand by Ohio regional
industry and to contribute to the diversity of the region’s work force. The project objectives include:
1. Establish a Mobile Application Development Technology 2+2+2 pathways featuring:
a. A model articulation agreement from high school to community college and then to
university level baccalaureate programs.
b. The expansion of the existing Industry Advisory Committee into a true BILT under the
guidance of Dr. Ann Beheler from the National Convergence Technology Center (CTC) so
industry co-leads work and provide maximum benefit for students.
c. A collaborative Mobile Design and Development certificate curriculum led by the BILT.
2. Create a collaborative state-of-the-art curriculum leveraging existing coursework in Interactive
Media and Computer Science:
a. An online curriculum featuring digitized, adaptive learning object methodology.
b. A project-based learning capstone supported by a scalable, state-of-the-art device lab that
that will provide the real-world work experience in designing and using mobile application
development technology-related software and hardware.
3. Recruit underserved, disadvantaged populations (defined as low-income, first generation students)
to the new SMARTT (Mobile Application Design and Mobile Application Development) pathways
through partnerships with high school districts, special interest programs, and non-profit
organizations.
Evaluation Questions
Formative Questions (FQ)
4. What activities, processes, and practices contributed to the establishment of the College’s Mobile
Application Development Technology 2+2+2 pathway?
5. How did the project facilitate leveraging the existing coursework to create a state-of-the-art
curriculum leading to an education program for mobile application development technicians?
a. Development of a collaborative, interdisciplinary capstone course focused on project-
based experiential learning.
b. Integration of project-based learning and soft skills development in all courses.
c. Approaches to faculty development strategy.
d. Delivery of blended curriculum.
e. Development and use of the SMARTT project device lab.
6. To what extent did the project recruit disadvantaged populations to the new SMARTT pathway?
Summative Questions (SQ)
1. To what extent has the project established a Mobile Application Development Technology career
education 2+2+2 pathway for a disadvantaged/underserved student population?
a. Establishing and executing model articulation agreements with high schools and
university level baccalaureate programs.
b. Developing and expanding curricula with industry input from the BILT, such as the
development of a new Collaborative Interdisciplinary Mobile Design and Development
(CIMDD) certificate.
c. The number of underserved and female students who:
i. Enroll in the new Mobile certificate
SMARTT Project Year 1 Evaluation Report 15
ii. Complete the new Mobile certificate
iii. Enroll in the Mobile AAS degree program
iv. Remain in Mobile AAS degree program
v. Complete capstone projects
vi. Enter the workforce after completing their certificate or capstone project
vii. Complete the Mobile AAS degree program
viii. Begin matriculation to a four-year institution.
2. To what extent are students in the Mobile Application Development Technology career education
2+2+2 pathways satisfied with the program?
Project Activities
These include:
• Complete industry and educational partner peer review of pathway and curriculum with the BILT.
• Create a new collaborative Mobile Application Design and Development certificate.
• Finalize transfer agreement with Franklin University to serve as a model articulation.
• Expand articulation with new high school and four-year partners.
• Develop a collaborative, interdisciplinary capstone course focused on project-based experiential
learning for students in the mobile application development technician pathway.
• Integrate two threads into all courses: project-based learning and soft skills development.
• Integrate a project-based learning thread into the existing mobile curriculum, providing the
opportunity to demonstrate application development and user interface design skills in a real-
world environment; students will design and create mobile applications coordinated within a
cloud-based project management platform for use by companies and non-profit organizations.
• Attend several Working Connections IT Faculty Development Institutes to learn how to become a
new hosting site.
• Create a scalable and replicable flagship “SMARTT Device Lab” housed on campus that
includes Android and iOS, phone and tablet devices for application testing.
• Promote program in large local school districts to encourage high-achieving low-income, first-
generation graduates to enter the SMARTT Pathway.
• Partner with U.S. Department of Education TRIO programs to identify and assist first- generation,
low-income students in the SMARTT pathway.
• Expand partnership with Per Scholas to provide seamless transfer of low-income adults from
CompTIA certifications into the SMARTT pathways.
• Gather follow-up records for capstone completers.
• Share and collect documents/records with four-year institutions for articulating students.
• Update student contact information for follow-up activities.
Project Benchmarks
Project managers will track program progress based on benchmarks outlined in Table B1. While the new
degrees and the certificate curriculum are designed during the project, many of the foundational courses
already exist, allowing enrollment of students in Year 2 as the second year classes are finalized.
SMARTT Project Year 1 Evaluation Report 16
Table B1. Project Outputs, Outcomes and Benchmarks
Project Component
Year
1
Year
2
Year
3
Total
Number of new companies engaged in BILT
16
2
2
20
Number of new students enrolled in new Mobile certificate
0
24
30
54
Number of new students continuing toward Mobile AAS degree
0
24
30
54
Semester-to-semester retention percent for students in program.
60%
65%
70%
70%
Number of students completing new Mobile certificate (launch Y2)
-
10
15
25
Number of students completing Mobile AAS degree (launch Y2, first graduates
Y3; others will graduate in sustainability period beyond project)
- - 8 10
Number of students articulating to 4-year institutions (starts Y3)
-
-
4
4
Number of faculty participating in Working Connections Institute
5
5
20
30
Number of capstone projects completed by students (one project per year per
student; Y3 includesY2 students (24) and Y3 students (30)
0 24 54 78
Benchmark: at least 90% of graduates express satisfaction with SMARTT
program
- 90% 90% 90%
Benchmark: at least 30% of students obtain employment after completing
certificate while finishing their associate’s degree
- - 30% 30%
Project Impacts (beyond grant period)
• Increase in retention and completion rates of students in SMARTT pathways
• Increase in number of disadvantaged and underserved students enrolled in SMARTT pathways
• Increase in number of technicians are educated and prepared to meet industry demand
• Faculty expertise is expanded in mobile technology
• BILT members hire and express satisfaction with SMARTT graduates
Evaluation Activities
As shown in the evaluation design table, numerous internal [I] and external [E] evaluation activities are
proposed.
Instrument Development
UCESC will work collaboratively with CCSC to develop relevant rubrics for all aspects of the evaluation
that need rubrics (e.g., assessing student projects, assessing quality of curricula, and assessing quality of
instructional apps). Some examples that may be used and/or modified by the team (with permission) are
the Course Evaluation Rubric for a Capstone Project in Design (Evenson, 2016) or the Evaluation Rubric
for Instructional Apps (Lee & Cherner, 2015). The final development will depend on the priorities and
decisions of the team; the process of rubric development detailed by Lee & Cherner (2015) is a best-
practice model.
UCESC has extensive experience developing protocols for focus group discussions and interviews as
well as surveys. As specified in the proposal, these instruments will be developed in collaboration with the
project team to ensure content and topics meet the needs of the project team and align with project
objectives and evaluation questions.
Course approval timing has allowed the evaluation team to establish a comparison student group for the
2018-2019 academic year with SMARTT curriculum changes being implemented beginning in fall 2019 to
assess student reactions to curriculum changes. The project and evaluation teams have no control over
when comparison students started in their program; thus, the evaluation team will administer a post-
SMARTT Project Year 1 Evaluation Report 17
survey only when students graduate, either at the end of fall 2018 or spring 2019 semesters. Instruments
to evaluate student experiences include:
1. Student self-efficacy survey
2. Student confidence survey
3. Employability and employment student survey
4. Program evaluation survey (administered at the end of capstone course)
In keeping with best practices in survey development, UCESC will conduct a literature review of existing,
standard measures used to assess similar topics using pre-established, existing surveys that have high
reliability and validity. Scales, subscales, and items from such pre-existing instruments will be adapted to
meet the needs of the project. Drafts of surveys will be shared with the project team for suggested edits
and feedback. Psychometric properties will be assessed for reliability and face validity. Faculty surveys
and protocols for student focus groups will be developed as the project progresses.
Data Collection
• The project team [I] will collect all relevant data for curriculum mapping and BILT feedback for
knowledge and skills assessment.
• The project team [I] will collect all relevant data for course evaluations and rubric scores.
• UCESC [E] will launch, field, and download student and faculty surveys.
• UCESC [E] will conduct all focus group discussions with students and BILT members; discussions
will be audiotaped.
Analysis
• UCESC [E] will check fidelity of curriculum mapping against BILT-defined knowledge and skills.
• UCESC [E] will thematically analyze focus group discussion data.
• UCESC [E] will conduct appropriate statistical analysis on survey and quantitative data.
References
Senior Interaction Capstone Project. School of Design, CMU. Pittsburgh, PA: Eberly Center for
Teaching Excellence, Carnegie Mellon University. Retrieved on June 12, 2017 from
https://www.cmu.edu/teaching/resources/Teaching/CourseDesign/Assessment-
Grading/Rubrics/DesignCapstoneRubric.doc
Lee, C-Y. & Cherner, T. S. (2015). A comprehensive evaluation rubric for assessing instructional
apps. Journal of Information Technology Education: Research, 14, 21-53. Retrieved from
http://www.jite.org/documents/Vol14/JITEV14ResearchP021-053Yuan0700.pdf
SMARTT Project Year 1 Evaluation Report 18
Project Goal: To develop an education program for mobile application development technicians at Columbus State Community College
to meet the increased demand by Ohio regional industry and contribute to the diversity of the region’s work force.
Evaluation Questions
[FQ = Formative; SQ = Summative]
Evaluative Activities
[I=Internal; E=External]
Timing
contributed to the establishment of the College’s
Mobile Application Development Technology
2+2+2 pathway?
• Review completed curriculum peer review by BILT [E]
Spring-Summer 2018
• Collaboratively develop a rubric to evaluate the Mobile
Application Development pathways [I,E]
Fall 2018
• Check mapping of course curriculum [E]
Fall 2018
• Review articulation agreements with Franklin University and
other high school and four-year partners [E]
Spring 2019 and 2020
existing associate degree programs to create a
state-of-the-art curriculum leading to an education
program for mobile application development
technicians?
This will be assessed in terms of
development of a collaborative,
interdisciplinary capstone course focused
on project-based experiential learning
• Collaboratively develop a rubric to evaluate the capstone
course [I,E]
Winter 2018
• Review CSCC course evaluations for all courses in
programs [I] and provide results to evaluation team for
analysis [E]
o Courses currently taught and significantly changed:
Programming Fundamentals for iOS (CSCI 1650),
Programming Fundamentals for Android (CSCI
1660), iOS Mobile Apps Development (CSCI 2650),
Android Mobile Apps Development (CSCI 2660),
and IMM 2372 course in Adobe Phonegap
Comparison 2018-19
Modified 2019-20
• Review developed capstone course for PBL elements [I]
Fall 2019
• Faculty survey: the collaborative and interdisciplinary
process of capstone course development [E]
Spring 2019 and 2020
SMARTT Project Year 1 Evaluation Report 19
Evaluation Questions
[FQ = Formative; SQ = Summative]
Evaluative Activities
[I=Internal; E=External]
Timing
existing associate degree programs to create a
state-of-the-art curriculum leading to an education
program for mobile application development
technicians?
This will be assessed in terms of
integration of project-based learning and
soft skills development in all courses within
a blended curriculum
• Summarize the process and final products of courses with
PBL and soft skills elements [I]
o Conduct performance assessments of student-
created and designed mobile applications [I]
o Review and describe the coordination of student
projects with cloud-based project management [I].
Comparison 2018-19
Modified 2019-20
• Review curriculum mapping to track integrations between
courses for both Mobile App Development and Mobile App
Design curriculum [E]
• Looking at
o Pre-requisites
o Knowledge and skills building on each other (PBL,
Soft skills, Application development, User interface
design development, Cloud-based project
management)
o
Ties to real-world environment
Fall 2018 (when
curriculum approved)
• Comparison of student performance assessments results
[E]
Comparison 2018-19
Modified 2019-20
• Conduct surveys to determine faculty members’ and
students’ perspectives about the MADT education program
and the new SMARTT pathway after changes, using 1) post
student survey for the comparison group and 2) pre-post
student surveys for the CSCC students [E]
Comparison 2018-19
Modified 2019-20
• Conduct pre-post surveys to determine CSCC students’
perspectives about the MADT education program and the
new SMARTT pathway [E]; conduct focus groups
discussions (FGD) with selected graduates
Modified 2019-20
FGD with graduates
existing associate degree programs to create a
state-of-the-art curriculum leading to an education
program for mobile application development
technicians?
• This will be assessed in terms of approaches to faculty
• Keep IT Institutes’ attendance records, review documents
related to IT Institutes and describes lessons learned from
IT Institutes [I].
Summer 2018, 2019,
2020
• Conduct faculty surveys to determine effects of faculty
members’ experience at the IT Institutes on the MADT
education program and the new SMARTT pathways
Fall 2018, 2019, 2020
SMARTT Project Year 1 Evaluation Report 20
Evaluation Questions
[FQ = Formative; SQ = Summative]
Evaluative Activities
[I=Internal; E=External]
Timing
existing associate degree programs to create a
state-of-the-art curriculum leading to an education
program for mobile application development
technicians?
This will be assessed by use of the
SMARTT Project Device Lab
• Collaboratively develop a rubric to evaluate the applicability,
scalability, and replicability of the “SMARTT Device Lab”
[I,E]
Fall 2018
• Conduct evaluation of the “SMARTT Device Lab” using the
developed rubric [I]
Beginning Winter
2019; assessed
annually
• Assess (via student and faculty surveys and student focus
group discussion) the applicability, scalability, and
replicability of the “SMARTT Device Lab” [E]
Modified 2019-20
FGD with graduates
disadvantaged populations to the new SMARTT
pathway?
• Document and summarize all promotional materials used in
recruitment [I].
Winter 2019 and
ongoing
• Document and summarize the project collaboration and
partnership with USDOE Trio program and Per Scholas [I].
Winter 2018 and
ongoing
• Check for project expected results regarding identification
of first-generation, low-income students in the program and
the transfer of low-income adults into the SMARTT
pathways [E]
Spring 2019 and
ongoing
Mobile Application Development Technology
(MADT) career education 2+2+2 pathway for a
disadvantaged student population in terms of:
Established and executed model articulation
agreements with high schools and baccalaureate
university level programs
• Review curriculum mapping and alignment with CSCC
official articulation documents provided by project team with
other educational institutions pertaining to course credits
allowed and instructor requirements of dual-level courses,
etc. [E]
Spring 2020
Mobile Application Development Technology
(MADT) career education 2+2+2 pathway for a
disadvantaged student population in terms of: The
expanded Building and Industry Leadership Team
(BILT) leading curricular development such as the
development of a new Collaborative
Interdisciplinary Mobile Design and Development
• Determine BILT involvement in the development of the
SMARTT curriculum and the CIMDD certificate (via focus
group discussion) with representatives of BILT [E].
Fall 2018
• Determine faculty perspectives (via faculty survey)
regarding the SMARTT curriculum, the CIMDD certificate,
and the 2+2+2 MADT career pathways [E].
Fall 2019 and 2020
SMARTT Project Year 1 Evaluation Report 21
Evaluation Questions
[FQ = Formative; SQ = Summative]
Evaluative Activities
[I=Internal; E=External]
Timing
Mobile Application Development Technology
(MADT) career education 2+2+2 pathway for a
disadvantaged student population in terms of: The
number of underserved and female students who
…
i. Enrolled in the new Mobile certificate;
ii. Completing the new Mobile certificate;
iii. Enrolled in the Mobile AAS degree
program;
iv. Retained in Mobile AAS degree
program;
v. Completed capstone projects;
vi. Entered the workforce after completing
their certificate or capstone project
vii. Completed the Mobile AAS degree
program;
viii. Began matriculation to a four-year
• Track student numbers in each program from entering
pathways through CSCC graduation
o Entering to completion of certificate
o Entering to completion of Mobile AAS degree,
including new capstone course
o Workforce participation during or after degree
completion
o Matriculation into four-year institutions
Ongoing
Application Development Technology (MADT)
career education 2+2+2 pathways satisfied with the
program?
• Conduct pre-post surveys to determine CSCC students’
perspectives about the MADT education program and the
new SMARTT pathways [E]
Spring 2020
• Conduct focus group discussion of selected students who
graduated about their satisfaction with the MADT education
program [E].
Spring 2020