|
Report of Progress in General Education Assessment Background This report fulfills the 2000 NCA Site Visit Team’s requirement that IPFW submit a report on progress toward strengthening its general education assessment process by June 2004. The team indicated that the assessment of student academic achievement in the general education component needs to be strengthened to include multiple measures of expected learning outcomes. The assessment should be used to improve the program including the periodic review of approved general education courses (Report of a Visit to IPFW, p. 59). The team had also noted that “the
objectives [of General Education] are not publicly articulated” (p.
19). The team therefore requested that IPFW submit “a progress report
on the assessment of general education that demonstrates progress toward
Level Two…” (p. 64) of the Assessment
Culture Matrix by
June 2004. Part I of this progress report reviews
the process we developed for assessment improvement. Part II describes
the changes we have made to enhance the institutional infrastructure
that supports school and department program assessment. Part III summarizes the specific steps we have
taken to strengthen general education assessment. Part IV reviews our plans for the future. Part I: Improvement Process IPFW implemented a university-wide
GE requirement in 1995. It is
essentially a distribution requirement unified by broad educational
goals (skills in communication, writing, quantitative reasoning and
critical thinking). Over the years, the program has undergone several
revisions, culminating, in 1999-00, in a thorough rewrite of the program’s
goals, with a twofold intent: the goals needed to be more transparent
to departments offering courses approved for the GE requirement as well
as more specific so that meaningful assessment would be possible.
This need to improve the GE assessment process was also one of
the concerns noted by the North Central Association Accreditation team. The revised GE program went into effect
in fall 2001. However, the General
Education Subcommittee (GES), a Senate subcommittee charged with overseeing
the program, was unable to agree on an approach to assessment. The reasons are not difficult to understand:
committee members tended to view general education primarily from the
perspective of their own department or school and had little knowledge
of assessment beyond the department level.
Therefore,
in the spring of 2002, the Vice Chancellor of Academic
Affairs appointed an Assessment Task Force (ATF) charged with the responsibility
of developing a plan for strengthening general education assessment. Members were selected to represent key constituencies
and Senate governing committees -- the Education Policy Committee (EPC)
and the General Education Subcommittee (GES). ATF members included Jeanette Clausen, Associate
Vice Chancellor for Faculty Affairs and Professor of German; William
H. Bruening, Associate Dean of Arts and Sciences
and Professor of Philosophy; Steven T. Sarratore,
Associate Vice Chancellor for Academic Programs and Professor of Theatre;
and Yvonne Zubovic, Associate Professor of
Mathematical Sciences. As a first step, ATF members
attended a working conference on assessment sponsored by the American
Association of Higher Education and the Higher Learning Commission in Part II: Strengthening the
Assessment Infrastructure In keeping with the tasks outlined in the Assessment Action Plan, the ATF
proceeded with the following tasks: 1.
Revise Senate Document for Assessment. The ATF led the process for gaining Senate approval
to revise Senate Document 98-22 The Plan for the Assessment of Student
Academic Achievement (SD 98-22 attached as Appendix B) to reflect
experiences with implementation of assessment, procedures of the Assessment
Council, and an update to the plan for general education assessment.
As a result, SD 98-22 now provides a comprehensive description of current
policies, procedures and services as a baseline for the continued improvement
of assessment at IPFW. Changes in SD 98-22, for example,
included increased flexibility in creating and implementing unit plans. Rather than mandating specific measures, the
document now specifies that “every assessment plan will systematically
utilize multiple measures of student learning, taken at multiple points
in time during an academic program, measuring multiple constituencies.” Many units, including Psychology, Communication,
Electrical and Computer Engineering Technology, Civil and Architectural
Engineering Technology, and Manufacturing Technology have changed their
plans to take advantage of this increased flexibility and generate data
of greater use to them in evaluating their programs. Another significant change involved
revising the reporting process so that the principal responsibility
for review of the unit report was delegated to the schools, with the
Assessment Council then reviewing school-level, rather than program-level,
reports. This change has had a significant and positive
impact on the campus assessment climate. For example, schools and departments now understand
that assessment is the responsibility of the faculty in the academic
units and that the Assessment Council is there to support, not police,
their work. As a group, faculty
are more comfortable with assessment as they realize its purpose is
not to make judgments about them as faculty; rather, it is a tool for
improving student learning in their programs.
An analysis of the school reports for the past year indicates
that faculty are beginning to discern the value
of direct measures over indirect measures and efforts are underway to
institutionalize appropriate direct measures at the program level.
Further, as data from direct measures of learning accumulate,
most schools’ beliefs are shifting away from engaging in assessment
in order to comply with institutional expectations to an appreciation
of assessment as a set of meaningful activities that provide insights
into their programs. Additionally, schools and programs utilizing
paper portfolios for student assessment are considering electronic versions
as a means of managing data more effectively and as a means of increasing
computer literacy among students. 2.
Restructure the Assessment Council. The Assessment Council
membership was redefined in the same revisions to SD 98-22 to require
that the membership consist of faculty or administrators responsible
for assessment in their respective schools and units. Although this
change appears subtle, it represented a significant shift in the functions
of the Assessment Council and in how the Council was perceived by the
campus community. Formerly composed largely of administrators and untenured
faculty members who evaluated assessment plans and reports, the Assessment
Council was perceived by the campus community as the “assessment police,”
dictating and enforcing unpopular policies.
This body is now made up of faculty with significant responsibility
for assessment in their academic units who review plans and act as assessment
advisors, providing feedback and support to departments and programs.
Currently, the Council includes the
Director of Assessment, the Associate Vice Chancellor for Academic Programs,
the chair of the General Education Subcommittee, a representative from
the Educational Policy Committee, representatives from Academic Counseling
and Career Services (ACCS) and General Studies, and one each from the
Schools/Divisions of Arts and Sciences; Business and Management Sciences;
Education; Engineering, Technology, and Computer Science; Visual and
Performing Arts; Health Sciences; and Public and Environmental Affairs. Importantly, the Council is no longer chaired
by an administrative appointee as it now elects its own chair from among
the faculty representatives. 3.
Create a Director of Assessment Position. In Spring 2003, the
VCAA authorized a new full-time position of Director of Assessment,
with responsibilities for coordinating and integrating campus general
education and program assessment activities. The position, which reports
directly to the Office of the Vice Chancellor for Academic Affairs,
was filled in October 2003 by an experienced assessment administrator.
The position profile is attached as Appendix C.
After only 10 months on the job, the Director of Assessment has
had a major impact upon the campus assessment climate, meeting with
countless academic units and committees, as well as individual faculty
and staff members to review or recommend effective assessment strategies.
She also analyzed and presented for
campus distribution data from Noel-Levitz Student Satisfaction
Inventory (SSI) and the National Survey of Student Engagement (NSSE). In addition, she has been integrally involved
with the development of an effective general education assessment plan
(see below). 4. Establish Institutional Assessment
Processes and Supports. One
of the major thrusts of the Assessment Action Plan was the development
of strong institutional assessment processes and supports. Numerous steps were taken in this direction
and are described below. a. The Assessment COQ: During the 2002-2003 academic year, the VCAA
developed a “Conversation on Quality” (COQ) initiative related to assessment.
The COQ program was created by the Office of Academic Affairs as a vehicle
for integrating and planning new and continuing activities designed
to enhance the quality of academic programs. It serves as a way to focus
attention and stimulate action on important issues of academic program
quality. The purpose of the COQ on assessment is to:
One significant activity of the COQ
was sponsoring two campus open forums on assessment on January 29 and
30, 2003. The forums allowed
faculty and others to air their concerns about assessment and assessment
processes and were instrumental in helping the Assessment Council shape
its recommendations for future assessment efforts and in establishing
the Director of Assessment position as a resource for faculty. Several key issues surfaced during the forum, including: "Assessment is formative; Program Review is summative." "If your assessment plan isn't useful to you, do something else that is." "Make assessment
work with accreditation." b. Assessment Mini-Grants: Feedback during the forums included support for a recommendation to create of a mini-grant program for faculty and departments requiring funding for assessment projects. As a result, almost $5000 in mini-grants has been awarded to departments over the last two years: · The Department of Communication to support departmental assessment committee members to travel to the IUPUI Assessment Conference. This resulted in significant revisions to the department’s assessment plans. · The Department of Psychology to support a temporary staff person to assist with the data collected from a survey of majors and alumni. This data played a major part in the department’s program review, resulting in revisions to their assessment plan as well as curricular changes. · The Department of Sociology to support a research assistant to conduct an alumni telephone survey. This data played a major part in the department’s program review. ·
The · The Department of Nursing to support a student assistant to enter student information into a database as part of a longitudinal assessment effort. This work is nearing completion, but results are not yet available. · The Department of Human Services to support a student assistant to enter student information into a database as part of a longitudinal assessment effort. This work is nearing completion, but results are not yet available. · The Department of Electrical and Computer Engineering Technology to support data entry for designing, plotting and analyzing a revised assessment measure. This work is nearing completion, but results are not yet available. c. On-campus Assessment Workshops: The Office of Academic Affairs and the Assessment
Council also continued to offer at least one campus-wide assessment workshop each academic
year in order to bring new ideas and resources to departments working
on assessment. The 2002-03 workshop was on
“Using Direct and Indirect Assessment Measures” and “An Introduction
to Electronic Portfolios” (both presented by Gloria M. Rogers of Rose-Hulman
Institute of Technology); the 2003-04 session was “Aligning General
Education, Departmental and Institutional Goals” (presented by Peggy
L. Maki, private consultant and formerly with the American Association
of Higher Education). The workshops were well attended with more than
80 participants at each, received positive evaluations and generated
discussions about general education and program assessment across campus.
The latter workshop supported the process of developing institution-level
expectations in the general education areas of quantitative reasoning,
oral communication and written communication. d. Assessment Conferences and Presentations: One of
the outcomes of heightened awareness of assessment at IPFW is the involvement
of faculty and staff in assessment-related presentations and research.
Faculty teams presented at an Association of American Colleges and Universities
conference on “Enlisting Faculty in General Education Assessment” in February
2003 in Philadelphia, and at Indiana University-Purdue University Indianapolis
Assessment Conference on “The Campus Assessment Council: What Works?”
in October of 2003. The Director
of Assessment and faculty in several departments are also engaged in
research designed to improve our understanding of the effects of IPFW
on student outcomes and as a form of scholarship. Many other faculty members have attended assessment-related
workshops at IUPUI, at Rose-Hulman Institute
of Technology, and at various other disciplinary or accreditation meetings. Assessment is evolving at IPFW. Many
programs, and in fact schools, have mature assessment processes with
multiple years of data being utilized for improvement. Others are actively
developing assessment plans and still others are in the initial stages
of learning about assessment and its value to their programs and students. We believe that our efforts to strengthen the
institutional assessment infrastructure are having a positive impact
….. Part III. An Effective General
Education Assessment Program/Process The second goal of the 2002 Assessment
Action Plan focused upon the development of an effective general education
assessment program. The Plan
proposed a process in which faculty teams would develop assessment strategies
for each of the six IPFW general education areas. The ATF decided very early not to raise any
questions about the content of the General Education program, although
certainly such proposals were in the air, but instead to focus entirely
on assessing the current program. If
the new assessment program worked as intended, program improvement would
naturally follow. (See Appendix
D for SD 99-25 General Education at
IPFW: General Education Program Definition and Course Criteria). The specific activities we undertook are described
below: 1. Develop General Education
Assessment Principles. The ATF’s Assessment
Action Plan first
reaffirmed the general education assessment principles that previously
developed as an update to the original SD 98-22 The Plan for the Assessment
of Student Academic Achievement and are as follows: ·
The assessment plan will be designed to measure
program goals, not the goals of individual courses. ·
To ensure a reasonable degree of consistency
across departments and schools, all courses to be included in the General
Education program will be approved by the General Education Subcommittee (GES)
based on the criteria approved by the IPFW Senate. To ensure continuity, the
GES will periodically conduct a review of approved courses.
·
The General Education Assessment Plan and its administration will be
consistent with the General Standards for Assessment Plans and with
2. Appoint and Charge General
Education Area Assessment Subcommittees. Next, the ATF appointed an
Assessment Subcommittee
for each of the six general education areas. Each Subcommittee included faculty teaching
in the area, a member of the Assessment Task Force and the Director of
Assessment. The Assessment Task
Force acts as a steering committee for
the activities of the subcommittees, working to coordinate and support their efforts. The subcommittees were charged to:
The proposed implementation schedule was for the
Area I–III subcommittees to plan during 2002–2003, supervise a pilot
during 2003-2004, and fully implement during 2004-2005. The Area IV-VI
subcommittees would follow the same development cycle a year later. 3. Evaluate current courses for
compliance with current GE standards. Each of the area subcommittees began by reviewing
syllabi collected from faculty teaching general education courses in
that area. They then developed
curriculum maps showing which general education goals were addressed
in which courses in order to evaluate the current status of the general
education program. The process
produced program improvements almost immediately.
For example, the area subcommittees reviewed the stated learning
objectives on each syllabus and found that although many of these objectives
were consistent with the goals of general education, the connection
to general education objectives was not always explicitly stated.
As a result, reminders are now sent to all faculty teaching general
education courses prior to each semester reminding them to include the
appropriate general education learning objectives on their syllabi. This has greatly increased compliance. The curriculum maps generally revealed
that courses were in compliance with the general education course criteria,
although some small discrepancies were found. Importantly, The Area II Subcommittee (Physical
and Natural Sciences) discovered that almost no courses were addressing
the objective for students to develop a familiarity with scientific
literature. Faculty discussions
are currently underway to decide how to revise this objective. The Area
V Subcommittee (Creative and Artistic Expression) felt that it was important
to add language to the objectives about the students’ use of an artistic
vocabulary. As a group, the area subcommittees have recommended that
the standing Faculty Senate General Education Subcommittee (GES) undertake
the systematic re-review of all current general education courses. The GES has agreed, and will initiate this process
during 2004-2005. It is anticipated
that the process will be implemented over a two to three year period.
4. Recast GE goals as learning objectives. In the summer of 2002, the ATF prepared an initial draft of general education learning objectives drawn from SD 99-25 General Education at IPFW: General Education Program Definition and Course Criteria recasting course criteria as student learning objectives. Each general education area assessment subcommittee reviewed and revised the draft learning objectives for its area during 2002-03. Now stated as learning objectives, these goals provide the foundation for the assessment plan and the ongoing process of review and program improvement. The revised objectives for the overall
objectives of the general education curriculum and the six areas are
listed below: IPFW General
Education Learning Objectives Upon completion of the IPFW general education curriculum,
learners will be able to: §
Identify substantive knowledge and disciplinary
methods; §
Gain and comprehend information; §
Use information; §
Analyze and synthesize information; §
Evaluate and assess their own and others’ ideas;
and §
Use knowledge and skills gained as a basis for
life-long learning. Area I: Linguistic
and Numerical Foundations Upon completion of Area I, learners will be able to: §
Speak and write precisely, clearly and
persuasively; §
Read and listen actively and with comprehension; §
Reason quantitatively as a means of gaining and
creating knowledge and drawing conclusions; §
Demonstrate appropriate information-gathering
skills; §
Apply the above skills in a range of contexts;
and §
Assess their own arguments and compare and
evaluate them with the arguments of others. Area II: Natural and
Physical Sciences Upon completion of Area II, learners will be able to: §
Describe bases of living and non-living systems; §
Explain development of scientific knowledge; §
Understand data collection, analysis and
quantitative problem solving; §
Demonstrate familiarity with scientific literature;
and
§
Apply their knowledge in written and/or oral communication. Area III.
Individual, Culture and Society Upon
completion of Area III, learners will be able to: §
Demonstrate an understanding of the nature and
diversity of individuals, organizations, cultures and societies;
§
Explain how knowledge of social and behavioral processes is developed,
how nformation is gathered, hypotheses formulated and analyzed, and
theories developed; §
Apply their knowledge in written and/or oral
communication; and §
Understand data collection, analysis and
quantitative problem solving. Area IV: Humanistic Thought (revisions
still in process) Upon
completion of Area IV, learners will be able to §
Demonstrate understanding of scholarly
approaches to such abiding issues as the meaning of life, the role of the arts
in understanding of what being human means and the limits of knowledge; §
Evaluate traditions that have shaped the
learners’ values, beliefs and aesthetic preferences; §
Compare traditions that have shaped the learner
to traditions different from their own; §
Interpret written work or creative work e.g.,
essays, works of art and music; §
Apply their knowledge in written an/or oral
communication; §
Understand data collection, analysis and
quantitative problem solving. Area V: Creative and Artistic Expression Upon
completion of Area V, learners will be able to §
Demonstrate an understanding of the creative
process using the vocabulary of the appropriate discipline; §
Perform or create a work of personal expression
and bring the work to fruition using applicable skills; and §
Articulate a reflective and critical evaluation
of their own and other’s creative efforts using written and/or oral
communication. Area VI: Inquiry and Analysis Upon
completion of Area VI, learners will be able to: §
Gather, evaluate, select, organize and
synthesize material in order to complete a research or creative project; §
Present the project in an appropriate medium; §
Think critically and solve problems by applying
knowledge and skills gained in earlier courses; and
§
Apply the knowledge gained across disciplinary boundaries. 5. Design specific measures and rubrics for assessing general education learning objectives. The ATF established several principles to govern the development of general education assessment measures, beginning with a commitment to direct measures of student learning. This marked a significant change from the previous general education assessment plan which included only indirect measures. Other principles are:
Based on these principles, each of the general education area assessment subcommittees designed measures and methods for assessing whether or not students were meeting the learning objectives for each area. After extensive discussion, the Area I-III committees are all following a similar process of collecting artifacts of student work and assessing these products against stated area learning objectives using a set of rubrics describing levels of proficiency. In general, the process proceeds as follows: 1. Each faculty member is asked to choose one written assignment (report, essay exam, analysis paper, etc.) from his/her course(s) that semester, and to submit a representative sample of the students’ work to the committee. 2. The students’ names are removed or concealed before sending the artifacts. 3. The photocopied artifacts are sent to the Assessment Director. The cover sheet asks faculty to identify the course and section number, to indicate which of the learning objectives are measured by the assignment, and to attach or reproduce the instructions for completing the assignment. 4. With each set of artifacts received, the committee will check off the course and section number from a master list of general education courses offered that semester. The purpose is to determine whether the committee has received student work from a representative sample of general education courses. 5. The artifacts are separated from the cover sheets and sorted into groups representing the various learning objectives. 6. After the end of the semester, a faculty team
assesses a random sample of the papers using a rubric describing levels
of proficiency. Findings are then reported to the Assessment Council
and published for information to the campus. The Area IV-VI subcommittees are considering
similar processes. Currently,
Area IV is developing a rubric and preparing a curriculum map simultaneously.
Areas V and VI reviewed syllabi and created assessment rubrics.
Student artifacts are being collected and pilot testing will be conducted
during 2004-2005. The progress made by each of the areas is consistent
with the timetable outlined in the ATF Action Plan. Although it is too early to draw firm conclusions or take actions based
upon the data generated by the first pilot studies, these pilots were
very useful in establishing the necessary processes, in helping to calibrate
those processes, and in establishing inter-rater reliability. The Area I (Linguistic and Numerical Foundations)
pilot seemed to point to a disconnect between
the learning objectives and the student outcomes in the required communication
course. Based upon this concern,
(and similar concerns in mathematics and English) the Area I objectives
became the focus of the 2004 IPFW Assessment Workshop with Peggy Maki
where groups of faculty and staff began a discussion of appropriate
learning objectives in these foundation skills.
Under Maki’s direction, the groups drafted learning objectives
and discussed assessment measures. This
project will be continued under the aegis of a new COQ (see below) during
the 2004-2005 academic year. 6. Promote General Education Awareness. To promote awareness of general education assessment and reinforce the area learning objectives, IPFW revised an earlier document and published FAQs Regarding Your Undergraduate Education at IPFW (Appendix E). The FAQ provides an overview of the purpose of general education, the learning objectives for each of the six areas and information on general education assessment. This document communicates IPFW’s expectations for student learning and introduces students to the role assessment plays in improving general education. It has been distributed to all incoming students at freshman orientation, to all faculty, and to advising centers in schools and departments across campus. Additionally, the assessment and general
education web sites (www.ipfw.edu/vcaa/Assessment/assmnthome.html
and www.ipfw.edu/academics/gened/
) were launched in 2003 and are constantly updated. The web sites are
linked and contain general education and assessment resources for students,
faculty and staff. All relevant
documents and policies on general education and assessment are on the
sites, including the new IPFW Assessment Handbook.
The handbook was produced by the Office of the Vice Chancellor
for Academic Affairs to promote and support continuous quality improvement
among programs and departments across campus.
It is an on-line resource designed for use by faculty, administrators,
staff, and students in academic program assessment and improvement. These steps directly address the Commission’s
concern about the public articulation of general education objectives. In summary, assessment of student
learning in general education is developing with the support of administration
and with the leadership of faculty and staff. Most importantly, faculty are involved in every phase of the assessment process.
We have learned that to be effective, general education assessment must
be driven by faculty, it must be decentralized, though with some coordination,
and it must be meaningful for faculty. Faculty have developed curriculum
maps, revised goals and objectives, identified measures, methods, and
appropriate metrics and have developed and piloted rubrics for general
education. Faculty are prepared to begin data
collection and evaluation in the upcoming academic year on schedule.
Perhaps most importantly, the process of designing a more complete general
education assessment process has prompted the campus to discussions
about general education and its effectiveness.
As we expected from the beginning, the issue of content quickly
follows assessment. Part IV. Future Assessment Activities Although much good assessment work
has already been accomplished at IPFW since the 2000 NCA site visit,
we recognize that we have much yet to do. General education assessment
will continue to proceed along the lines outlined in the ATF’s
Assessment Action Plan, and as data are generated, result in evaluation
and dialog about what they mean, how they should best be used and what
changes should be made to improve student learning. Indeed, discussion
has already begun about the effectiveness of the general education curriculum
and questions have been raised about the possible need for change based
on the process of designing an assessment plan. One immediate result is a new COQ
initiative, “The Common Elements of an IPFW Baccalaureate Education“
that will be launched by the Office of Academic Affairs in the fall
semester of 2004 as a vehicle for reviewing and evaluating undergraduate
learning at the institutional level. The purpose of the initiative is
to identify common themes or learning objectives expected of students
across all programs and general education.
A next step will be to determine if these
common learning golas are being realized and
to form action strategies to improve student learning as needed. The initiative has the potential to integrate
assessment, program review and institution-level benchmarks for student
learning. Other assessment activities planned for 2004-05 include:
IPFW has had considerable success implementing both the infrastructure and general education strategies of the Assessment Action Plan developed by the Assessment Task Force in 2002. We have reviewed our progress against the characteristics in the Assessment Culture Matrix and feel that we have attained Level Two in all points save II.c. Shared Responsibility: Students. We are extremely pleased with this progress, but we recognize that it is now time to draft a new action plan to move us beyond Level Two. The new plan will build upon our successes in assessment infrastructure and general education, assuring that these gains are institutionalized, and will also focus upon developing strategies to actively involve students in assessment. We look forward to continuous improvement as we celebrate the institution’s 40th anniversary. Appendix A: Assessment Action Plan Appendix B: Faculty Senate Document 98-22 The Plan for the Assessment of Student Academic Achievement Appendix C: Assessment Director position profile Appendix D: Faculty Senate Document 99-25 General Education at IPFW: General Education Program Definition and Course Criteria Appendix E: FAQs Regarding Your Undergraduate Education at IPFW |