Sociology Third Year Review Report - May 2008

Evaluation Rubric

Since our last third-year review in 2005, we have made substantial changes to our departmental assessment process. These changes were a direct result of the workshop with Barbara Walvoord held on campus in 2005 and of our attempts to make the assessment process more useful for our program. Copies of the 2005-06 and 2006 Assessment Reports are attached; the 2007 Assessment Report will be available by May 1, 2008.

Changes to educational goals: Since our last third-year review, we have added a sixth educational objective to our goals for sociology majors.  The previous goal was “Development of a multicultural perspective on the social world, including an awareness of the factors contributing to social and cultural diversity, and an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.” (We determined as well that by “multicultural” we also meant “global.”) Because past assessments had indicated that students were not achieving the global/multicultural aspect of the goal, the two goals were split to reflect this finding and to enhance our ability to focus on this specific aspect of our goals. Now they read: “Development of an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.” and “Development of a global and multicultural perspective on the social world, including an awareness of factors contributing to social and cultural diversity.”

Changes to assessment techniques: We have made a variety of changes to our student portfolios over the years. Currently, instructors are to place an example of student work from theory, methods and any other sociology courses taken after the major is declared in the portfolio. (We decided that materials from theory, methods and 400-level seminars did not provide sufficient information for analysis.) Unfortunately, this process is far from foolproof. Despite reminders from the department chair, not all faculty members place work from their courses in the portfolios. Students who take these courses before declaring sociology as a major or who take courses at other campuses have incomplete portfolios. In practice, we also include student transcripts in the portfolios to provide access to more information (course grades, which are not always recorded by advisors, and other courses taken that might be relevant to our goals, but that were, for example, taken elsewhere). We have developed a new rubric for portfolio assessment in an attempt to make assessment of the portfolios by multiple raters more valid and reliable (see link in Assessment Reports below.) After relying for years only on this direct assessment technique, our new assessment plan calls for an additional indirect assessment technique that surveys graduating seniors about their experiences in the program (see attached survey). The attached assessment reports reflect summaries of the assessment data. Raw data from previous portfolio assessments is now kept in the departmental assessment file. This updated Assessment Plan is attached below.

Conclusions from assessment: After discussing findings from the 2005-06 graduates, the department decided to focus on two concerns. Students were not meeting the global/multicultural educational goal so we needed to figure out how to improve that. We also needed to clarify the role of our 400-level seminars in light of questions about providing students with a “capstone” experience. Since our external program reviewer (and the American Sociological Association) also recommended such capstone experiences, we needed to decide how to best serve students in regard to that element of the major.
Lack of responses to the first attempt at a survey of graduates also raised the question of the usefulness and feasibility of that assessment technique. We decided to do one more year (2007) of the survey using better survey research methodology (e.g., sending the questionnaire out more than once, making sure we had correct addresses, etc.). An attempt to do a better job of soliciting responses to the graduate survey was our other focus of attention for 2007. (Note that because we had only one other 2006 graduate, no discussion of assessment was conducted in the department, as 2006 was our year to transition to a calendar-year approach to assessment. Unfortunately, the 2007 assessment process has taken longer than it should have, preventing us from discussing it in a spring 2008 department meeting as planned. It will be part of our departmental retreat on September 5, 2008.)

Changes made to program: As part of an overall reconsideration of the sociology major, we adopted guidelines for 400-level courses to ensure that these courses can function as capstone experiences despite their diversity of teaching style, topic, etc. Seminars are now expected to including the following elements: (1) substantial synthetic/analytical writing, (2) substantial reading with a focus on primary sources (i.e., articles and monographs). (Textbooks can play a supplemental role.) (3) Organization as a seminar, with less lecturing and more discussion/student participation. Instructors are encouraged to incorporate student presentation of readings, research findings, etc. (4) Student application of theories to course content.
This discussion of the major and of capstone experiences all led to the decision to offer a new kind of seminar, one in which students would have the opportunity to undertake a substantial research project under the guidance of a faculty member. This course is being offered for the first time in Spring 2008. While there were no specific assessment data that supported a reconsideration of our 400-level seminars, it was the assessment of the student portfolios that led to a discussion of the content and format of the seminars.
Because of other issues facing the department, specifically personnel shortages, we have chosen not to implement the revamped major yet. We have also been distracted by these other issues from a further consideration of the global/multicultural educational goal. There has been some discussion of how we might ensure that students develop this perspective.
We discussed adding a requirement that students take some course designated as having a global perspective or that certain courses always include such content. No final decision was made about what to do. This particular concern was a direct product of the assessment of major portfolios. We failed to see widespread evidence that students were being exposed to coursework that would help them develop a global/multicultural perspective. Some students had much exposure evident from their transcripts, for example; but other students seemed sorely lacking in this exposure.

Anticipated changes to assessment:  Having broadly revamped our approach to assessment three years ago, we do not anticipate large-scale changes in the near future. Doing a better job of placing materials in the portfolios will improve our data. Direct admission of students to the program may also help, as it means we will know right away that students are majors and can collect materials immediately. 
Because of the low response rate to the second iteration of survey of graduates, this spring (May 2008 grads) we will begin conducting the survey with students who are graduating rather than hoping they will respond to a survey sent to them after they graduate. (Despite checking to make sure we had current addresses and sending the survey out twice, our response rate remains low and makes it difficult for us to have reliable data to analyze.) The chair of the assessment committee will ask students to come in to fill out the survey before the end of the semester in which they graduate in an attempt to improve the response rate. While the response rate was better in the second iteration of the survey, it remains below 50 percent.
Finally, when the Anthropology major is approved, as is expected in the next couple of years, the department will need to implement an assessment of that program.

Involvement by faculty, students, administration, alumni and other groups: The Assessment Committee (3 faculty for 2-year terms with the current chair serving for 4 years as of 2008) is directly involved in assessment. The rest of the faculty is indirectly involved: an internal version of the assessment report is given to them; we discuss this report and decide what our focus of attention for the next year will be. In other words, all full-time faculty members have the opportunity to be part of the assessment process. Students are involved when they graduate and are asked to fill out the survey. Since faculty put the portfolios together, students are only indirectly involved in that aspect of assessment.  Assessment data have not been shared with the administration. Assessment results are shared with the IUSB Assessment Committee.

Summary:  The assessment process in our department has changed quite a bit in the past three years. For example, the development of a rubric for assessment of the portfolios facilitates the process of comparing faculty members’ individual assessments of the portfolios. This makes it easier to identify patterns in the achievement – or lack thereof – of educational goals, which allows us to better gauge whether students are meeting the goals. Once we determine that there is a lack of attainment, we can mobilize the faculty to determine how we can better serve students. As another example, the implementation of a survey of graduates gives students a direct role in the assessment process. It also gives us a better sense of what students think they’re getting out of the major, rather than relying only on our assessments of whether they have met our goals. If students completing our major do not think they are meeting our educational goals, then we can mobilize to make changes in the program to improve student learning, knowing that the students themselves have identified a weakness in the major. Once we determine a more successful mechanism for administering this survey, we will have better assessment data and can continue to improve our major program.

Assessment Plan for Department of Sociology and Anthropology
(adopted October 2005)
LEARNING GOALS
By the time students complete the major program in sociology, we expect them to demonstrate:

1. Ability to apply the sociological imagination to trace the links between individual experiences and social forces and between social forces and history to critically understand oneself.
2. Understanding of substantive issues, which include how culture and social structures operate, the reciprocal relationships between individuals and society, the relationships between macro and micro levels of social reality, the impact of social institutions and social inequality on society.
3. Understanding of sociological research methods, both quantitative and qualitative, including how to interpret findings of such research, awareness of the assumptions built into various research methods and how the choice of a particular method will may affect research conclusions and the advantages and disadvantages of applying a particular research method s to a research question.
4. Awareness of the role of sociological theory and ability to discuss, apply, and describe some basic theories or theoretical orientations in at least one area of social reality.
5. Development of a multicultural perspective on the social world, including an awareness of the factors contributing to social and cultural diversity, and an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.

ASSESSMENT MEASURES
Direct: Review of student portfolios of work from theory, methods and 400-level seminar courses by assessment committee.
Indirect: Survey of graduating seniors that asks how well they achieved departmental learning goals, what aspects of their education helped them with their learning and why, and what the department might do to help them learn more effectively and why those changes would help them.
ANNUAL DISCUSSION
Members of the department will meet each year to discuss the findings of the assessment committee’s review of student portfolios and of the survey of graduating seniors.  From this discussion, each year we will identify two action items.

Survey of Recent Graduates, Department of Sociology and Anthropology, IUSB
Please circle the response that best reflects your experiences as a sociology major.
Use these categories for your responses:
1                                                      2                                                     3
      Not as well                                       As well as                             Better than
     as I expected                                      I expected                       I expected
1. How well did you achieve the following departmental learning goals?
Ability to apply the sociological imagination to trace the links between individual experiences and social forces and between social forces and history to critically understand oneself.
1                                               2                                                                            3
Understanding of substantive issues, which include how culture and social structures operate, the reciprocal relationships between individual and society, the relationships between macro and micro levels of social reality, and the impact of social institutions and social inequality on society.
1                                                 2                                                                          3
Understanding of sociological research methods, both quantitative and qualitative, including how to interpret findings of such research, awareness of the assumptions built into various research methods and how the choice of a particular method may affect research conclusions and the advantages and disadvantages of applying a particular research method to a research question.
1                                                 2                                                                          3
Awareness of the role of sociological theory and ability to discuss, apply, and describe some basic theories or theoretical orientations in at least one area of social reality.
1                                                  2                                                                         3
Development of an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.
1                                                  2                                                                         3

Development of a global and multicultural perspective on the social world, including an awareness of factors contributing to social and cultural diversity.
1                                                   2                                                                        3
2. What aspects of your education in this department helped you with your learning?

Why were these aspects of your education helpful?

3. What might the department do differently that would have helped you learn more effectively?
Why would these actions have helped?
4. Other comments you would like to share with the department.

THANK YOU FOR YOUR ASSISTANCE!

NOTE: Survey is sent out with a letter from the chair of the department’s Assessment Committee asking students to complete and return the survey and assuring them that we are interested in assessing the major not individual students.

2005-06 Assessment Report
Department of Sociology and Anthropology
a. Program name: Department of Sociology and Anthropology
b. Report prepared by: Betsy Lucal, Associate Professor of Sociology; Chair, Assessment Committee
c. Current assessment contact: Betsy Lucal
d. Yes, assessment information can also be sent to Dan Olson, department chair

1. Our educational goals for sociology majors are for them to develop:
a. Ability to apply the sociological imagination to trace the links between individual experiences and social forces and between social forces and history to critically understand oneself.
b. Understanding of substantive issues, which include how culture and social structures operate, the reciprocal relationships between individual and society, the relationships between macro and micro levels of social reality, and the impact of social institutions and social inequality on society.
c. Understanding of sociological research methods, both quantitative and qualitative, including how to interpret findings of such research, awareness of the assumptions built into various research methods and how the choice of a particular method may affect research conclusions and the advantages and disadvantages of applying a particular research method to a research question.
d. Awareness of the role of sociological theory and ability to discuss, apply, and describe some basic theories or theoretical orientations in at least one area of social reality.
e. Development of an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.
f. Development of a global and multicultural perspective on the social world, including an awareness of factors contributing to social and cultural diversity.
The department previously combined goals e and f into one goal. However, previous years= assessment indicated that students were not achieving the global/multicultural aspect of this goal, so these two were separated to reflect this issue.
2. Our assessment techniques for 2005-06 were:
Direct: Review of student portfolios of work from theory, methods and 400-level seminar courses by assessment committee.

Indirect: Survey of graduating seniors that asks how well they achieved departmental learning goals, what aspects of their education helped them with their learning and why, and what the department might do to help them learn more effectively and why those changes would help them.
In the past, the department relied on just the direct assessment of portfolios of student work. Based on the recommendations of B. Walvoord, this year we attempted two assessment methods, adding a survey of graduating seniors (August and December 2005 and May 2006).
We also developed a rubric for assessing the student portfolios. It can be viewed at:
http://landmark-project.com/classweb/tools/printable.php?rbrc_id=166817
3. Results of this year=s assessment were discussed by the Assessment Committee (Otis Grant, Mike Keen and Betsy Lucal) and reported to the department on September 8, 2006. Below are some highlights of this discussion.
The faculty discussed the lack of responses from the mailed survey of alumni (no responses were received). Betsy will follow up by email with students to see what happened. We may need to consider doing exit interviews with majors to ensure they have an opportunity to respond to the survey.
We discussed whether we should examine student transcripts in addition to their portfolios. We decided that transcripts include information that portfolios do not: e.g., evidence of completion of a minor in anthropology or international relations that would show students had met Objective 6 through other channels. We discussed the new rubric and the continuing challenges of operationalizing evidence of students meeting objectives. For example, is taking and passing courses on two different kinds of inequality enough to show students have met Objective 5? Should course grades be taken into consideration? What is appropriate evidence of students meeting Objective 1? This process will continue to be refined by the departmental assessment committee. We also decided that we really do need materials from other courses (not just required – theory and methods – and capstone – 400-level seminars) to provide better evidence of whether students have met learning objectives. Again, with Objective 5, including materials from inequalities courses would be very helpful. We discussed the lack of evidence that students had met Objective 6. We discussed the issue of the relationship between a capstone experience and our 400-level seminars.
Finally, we discussed whether it would be more appropriate for the committee to do its work on a calendar-year schedule rather than an academic-year schedule. This would allow us to submit the report to the Assessment Committee in a timely manner.

4. Two concerns that the department has decided to focus on for the 2006-07 year are:
                - Objective 6: How can the department provide more opportunities for students to meet this goal?
                - Capstone/400-level seminars: If these seminars are going to continue to function as the equivalent of a capstone course, then the department needs to make some decisions about their form and content.

2006 Assessment Report
Department of Sociology and Anthropology
**Note: In 2006, we decided, with permission from the Assessment Committee, to move to a calendar-year assessment schedule rather than an academic-year schedule. In order to move to that schedule, this report adds only December 2006 graduates (N=1). In the future (beginning 2007), we will assess May, August and December graduates in our annual report. We believe this schedule will allow for a more timely discussion of assessment results (during a Spring semester faculty meeting) to determine goals for the following academic year.  It will also facilitate timely submission of our Assessment Report to the committee.
This report provides an update on our progress with respect to the two areas of emphasis for the 2006-07 academic year. We will continue to pursue these concerns until the 2007 report is made, at which time we will determine which concerns will be our next focus.
a. Program name: Department of Sociology and Anthropology
b. Report prepared by: Betsy Lucal, Associate Professor of Sociology; Chair, Assessment Committee
c. Current assessment contact: Betsy Lucal
d. Yes, assessment information can also be sent to Gail McGuire, department chair (beginning July 1, 2007)

1. Our educational goals for sociology majors are for them to develop:
a. Ability to apply the sociological imagination to trace the links between individual experiences and social forces and between social forces and history to critically understand oneself.
b. Understanding of substantive issues, which include how culture and social structures operate, the reciprocal relationships between individual and society, the relationships between macro and micro levels of social reality, and the impact of social institutions and social inequality on society.
c. Understanding of sociological research methods, both quantitative and qualitative, including how to interpret findings of such research, awareness of the assumptions built into various research methods and how the choice of a particular method may affect research conclusions and the advantages and disadvantages of applying a particular research method to a research question.
d. Awareness of the role of sociological theory and ability to discuss, apply, and describe some basic theories or theoretical orientations in at least one area of social reality.
e. Development of an understanding of social structures and processes connected with race, class, gender, sexuality, religion, etc.
f. Development of a global and multicultural perspective on the social world, including an awareness of factors contributing to social and cultural diversity.
The department previously combined goals e and f into one goal. However, previous years= assessment indicated that students were not achieving the global/multicultural aspect of this goal, so these two were separated to reflect this issue.
2. Our assessment techniques for 2006 were:
Direct: Review of student portfolios of work from theory, methods and 400-level seminar courses by assessment committee.
Indirect: Survey of graduating seniors that asks how well they achieved departmental learning goals, what aspects of their education helped them with their learning and why, and what the department might do to help them learn more effectively and why those changes would help them.
We also developed a rubric for assessing the student portfolios. It can be viewed at:
http://landmark-project.com/classweb/tools/printable.php?rbrc_id=166817
3. Results from the 2005-06 assessment activities allowed the department to identify two areas of concern to be the focus of our attention during the 2006-07 academic year and during Fall 2007. These concerns were:
            - Objective 6: How can the department provide more opportunities for students to meet this goal?
- Capstone/400-level seminars: If these seminars are going to continue to function as the equivalent of a capstone course, then the department needs to make some decisions about their form and content.
After identifying the capstone/400-level seminars as an area of concern (it was also an area of concern identified by our external reviewer, Nancy Greenwood, during her visit to our department), we spent time at multiple department meetings discussing how to respond to this issue. Betsy Lucal attended a workshop on capstone seminars, did some research on them (using materials prepared by the American Sociological Association) and reported back to the faculty what she found. Chair Dan Olson informally surveyed LAS departments offering capstone courses to see how they organized them; he reported his findings to the faculty.
As a result of these deliberations, the department voted to adopt a set of guidelines for 400-level seminars that will ensure that they can function as a capstone experience for our majors despite their diversity (of teaching style, topic, etc.).

Seminars are expected to include these elements:
1. Substantial synthetic/analytical writing.
2. Substantial reading with a focus on primary sources (i.e., articles and monographs). Textbooks can play a supplemental role.
3. Organization as a seminar, with less lecturing and more discussion/student participation. Instructors are encouraged to incorporate student presentation of readings, research findings, etc.
4. Student application of theories to course content.
Our discussion of capstone/400-level seminars also became part of a broader discussion about the restructuring of our major. As a result, we decided to add a new type of seminar to our 400-level offerings, one that would provide students with the opportunity to undertake a substantial research project under the guidance of a faculty member. While we have decided to hold off on implementing these broader changes to the major (because of personnel shortages facing the department), Gail McGuire will offer this new seminar course for the first time in Spring 2008.
Our focus on this concern, in addition to other issues facing the department, meant that we did not get to address the concern about the lack of development of a multicultural and global perspective by our majors.
Note: Because we had only one December 2006 graduate, no further meaningful analysis of assessment data was possible. With only one graduate, it would be impossible to assess the program rather than that student. So analysis is deferred until the end of 2007.
4. Two concerns that the department will focus on for 2007 are:
- Objective 6: How can the department provide more opportunities for students to meet this goal?
- Better implementation of our indirect measure (survey of graduating seniors). We received just one response to the survey (several months after it was sent). It is clear that we need to do a better job with this technique. The department assessment committee (Betsy Lucal [Chair], Gail McGuire and Jay VanderVeen) will discuss ways to get a better response rate and make a recommendation to the department.