Public Affairs - Graduate Third Year Review
Written Report - May 2007

Assessment Contact Person:     Barbara Peat

Evaluation Rubric

Assessment Plan

Describe any assessment techniques used for measuring Educational Goals that have been added or discontinued since the last Third Year Review

I became the Director of the Graduate Program in July of 2006, replacing Paul Herr who retired.  The Graduate Plan was recently developed.  Although an annual report should have been submitted in May of 2006 I do not find any records to indicate that one was done.  Neither were there any prior third year reports.  Therefore, I provide the following information based on the plan that was submitted in 2005 and do not have any other documents to supplement this information.

Assessment instruments used during the past three years
The instruments used in the capstone course (V600) are attached with the Plan.  The class is offered only in the spring of each year.  We will administer the same instrument before the end of the Spring 2007 semester and an addendum providing a summary of that data will be provided at the MPA presentation.  Students are given three options to choose from in responding to directed questions about their knowledge and skill improvements as an outcome of completing the MPA Program:  (1) no changes, (2) improved, and (3) improved significantly.  The following represents a summary of the data collected in the spring capstone course for the past couple of years. 

Spring 05 V600 Capstone Course – Student Survey:

11 respondents
The vast majority indicated improvement in:

• communication skills (8)
• ability to apply management theory and principles (6)
• awareness of legal constraints within which leaders and managers operate ability to apply theory (8)
• critical thinking skills (9)
• ability to understand and adapt to the ethical, social, economic, and political environment of public policy (7)
• problem solving skills (9)
• understanding of professional ethics (9)

The weakest areas were noted (through “not changed” responses) in:

• awareness of the impact of globalization on public management (3)
• awareness of economic constraints within which leaders and managers operate (3)

The strongest areas were noted (through “improved significantly” responses) in:

• quantitative skills (6)
• ability to apply management theory and principles (4)
• expertise in policy and management in my MPA area of concentration (5)

In response to a query about the reason why they felt a particular skill/knowledge was “not changed” through the education provided in the program the response most frequently given was “lack of curriculum emphasis”.

Spring 06 V600 Capstone Course – Student Survey:

13 respondents
The vast majority indicated improvement in:

• communication skills (10)
• problem solving skills (10)
• understanding of professional ethics (11)
• awareness of the impact of globalization on public management (11)

There were no significantly weak areas noted (with at least 4 responses)

The strongest areas were noted (through “improved/increased significantly”) in:

• ability to apply management theory (5)
• awareness of legal constraints within which leaders and managers operate (5)
• quantitative skills (5)
• critical thinking skills (6)
• ability to understand and adapt to the ethical, social, economic, and political environment of public policy (6)
• expertise in policy and management in my MPA area of concentration (6)

In response to a query about the reason why they felt a particular skill/knowledge was “not changed” through the education provided in the program the response most frequently given was “lack of curriculum emphasis”.

An alumni survey (appendix 2) was administered in 2004.  The survey was not only directed at graduates of the MPA program but was sent to all SPEA alumni who graduated between 1999 and 2003.  A total of 325 surveys were mailed.  Approximately 75 surveys were returned as “undeliverable”.  We received a total of 56 responses representing a response rate of 22.4% with 17 respondents indicating they graduated with from the MPA program.  Unfortunately, the person who compiled the responses did not break the data analysis down based on degree.  Therefore, it would not be beneficial to provide responses to specific questions because we are unable to separate out the MPA graduate responses from the rest. 

What analysis has been done with this data?  What conclusions has your department drawn?  What changes have been made to the program as a result?
I recently compiled the results from the capstone surveys.  No faculty discussion of the results has been conducted at this time.  However, the results will be an agenda item for discussion at the annual fall retreat in August 2007.  We will determine at that time if any part of the existing assessment plan needs to be changed and we will report those changes in the annual report submitted spring 2008.

How did assessment data and analysis support these changes?
Not applicable

What changes does the department plan to make in the coming years to the program and to assessment techniques, and why?
As the Graduate Director I plan to develop an alumni survey that gathers data from the MPA graduates separate from all other programs offered through SPEA.  In this way the results of the survey will provide focused information that can be used specifically for curriculum changes and program planning in the MPA program.  Secondly, I will be working on revising the existing capstone survey to gather more useful information about a broader range of topics than is currently covered.  Third, I will develop an employer survey to gather data from agencies in the area that have historically hired our MPA graduates.  Due to the fact this will be a totally new assessment strategy used for the MPA program I am likely to begin with interviews with a sampling of agency employers.  From that information I will be better prepared to develop the survey to be distributed to other agencies.  A fourth possibility I am exploring is to use a course embedded assessment strategy based on course objectives for each one of the required core courses (8 total).  This strategy would entail the development of a pre/posttest for each of the 8 core required courses based on the course objectives as listed in the course syllabi.

How were faculty, students, administration, alumni and other groups involved in assessment?
Faculty has been involved in assessment in their role of completing a survey at completion of the capstone course (V600).  A copy of the current instrument used is attached to the MPA assessment plan.  I have data collected from spring of 2006 using this method with plans for additional data to be collected from the faculty member of the spring 2007 capstone course before the end of the semester.
Students have been involved through their completion of the capstone survey.
Other students in the program (12 representatives from the current student body) are involved in assessment through their participation on the Graduate Advisory Committee.  At each of the meetings I have had with the committee during the 2006-07 academic year (a total of 3 meetings) I have queried them to provide me information about the “ins and outs” of the Program and to provide me with feedback they have received from other students in their classes.  Although an informal assessment strategy I have found it very useful to curriculum planning.
Alumni were sent a survey in 2004, as mentioned previously in this report.

How were assessment data and results shared with faculty, students, administration and alumni?
To my knowledge the results of the capstone course survey (2006) were not shared with other faculty but were shared with the dean.  The alumni survey results were shared with the entire SPEA faculty.  I have no knowledge of assessment results being shared with students, other administration representatives, or alumni.  This is an area we will have to improve upon in the coming year.

In one paragraph, please summarize the most important impacts of the assessment of student learning on the program
As is evident in the information provided throughout this report, we are in the beginning stages of getting the assessment protocol for the MPA Program up and running.  Therefore we are not at the stage where we have explored the impact of assessment on the Program.  However, this is an area that will be targeted in the coming year.

Supporting Information

Spring 07 V600 Capstone Course – Student Survey:

9 respondents
The vast majority indicated improvement in:

• communication skills (7)
• problem solving skills (6)
• understanding of professional ethics (8)
• awareness of economic constraints within which leaders and managers operate (6)
• critical thinking skills (9)
• expertise in policy and management in my MPA area of concentration (8)

There were only two areas that had two respondents indicating “not changed”: 
(1) communication skills in speaking and (2) ability to understand and adapt to the ethical, social, economic, and political environment of public policy

The strongest area was noted (through “improved significantly”) in:

• ability to apply management theory and principles (5)

Comparative analysis of instructor rating of student improvements:
The instructor of the V600 capstone course completes a review of the overall preparation level of the students in the class using the same criteria in the students’ self-rate survey.  I did a comparison of three consecutive years of reports (2005-2007).  In summary, the instructors rated the students’ preparation as adequate in most of the 11 criteria used in the survey.  Two areas were indicated by two of the instructors as “not evaluated” and those were “student awareness of legal constraints within which leaders and managers operate” and “student quantitative skills” (one instructor listing it as adequate, one as inadequate and one as not evaluated.  These are two areas that the faculty will have to meet, as a group, to explore and determine whether they are areas that should be covered in the capstone course.  If not, then these criteria should be removed from the survey.  One of the instructors listed the three areas of communication (writing, speaking, and listening) as more than adequate.  It is interesting to note that for both spring 2005 and 2006 students indicated that their quantitative skills improved significantly.  I am thinking that the difference stems from the fact that students self-rate based on the entire IUSB experience whereas the instructor rates on what is assessed in the capstone course.  We will need to explore better ways for the instructors to assess student learning outcomes.

 top