IU South Bend CAMPUS ASSESSMENT COMMITTEE
2000-2001

Project Overview:
Assessment Test Deveopment:
Test Implementation and Data Evaluation:
Future Work:
Budget:
Conclusions:

Computational Proficiency through Assessment - Project Summary

David R. Surma and James Wolfer
Department of Computer and Information Sciences

Submitted to the IU South Bend Assessment Committee

1. Project Overview

This project was undertaken to fulfill a need identified by the faculty of the department of Computer and Information Sciences. Specifically, the task was to develop a tool that could be used to both assess how well graduates of the program are prepared to enter the workforce or to carry on with graduate education and also to help identify deficiencies in the backgrounds of new students entering the department's graduate program. The IU South Bend assessment committee supported this project by funding the development of this tool, a comprehensive assessment examination, and this report summarizes work done in developing and administering the exam. Additionally, analysis of the results from the first use of the exam is presented along with a discussion of future plans for this tool.

2. Assessment Test Development

Work done on developing the assessment test was divided into two parts. One part dealt with the development of the test questions while the other part was concerned with the actual testing environment. The two principal investigators (PIs) for the project worked on the former and wrote questions to perform the following functions:

- Evaluate assimilation of foundation knowledge
- Evaluate computer programming ability

To meet these objectives it was decided to have one portion of the test be composed of multiple choice questions while the other portion would require students to write some type of program. Multiple choice questions (and other question ideas) were solicited from faculty members in the department and recent GRE computer science questions were reviewed. Additionally, new questions were created to help test knowledge stemming from five of the department's core courses. When creating the questions a two-hour time allotment was agreed upon. For the first version of the test, 16 questions were selected with additional questions given to students who had taken various electives in the department. The second portion of the test required students to create a portion of a computer program written in either C or C++. The type of program written was a class, which is a software entity that contains functions used in solving some application. Students were instructed to write a class for the stack data structure. This problem was chosen since students should be very familiar with stacks and most of them would have written programs dealing with them in at least one prior course.

The second task in developing the test was the testing environment, or platform. It was decided that the test should have a web-based interface. Students would go to a specified URL and would be able to access the test in simple HTML format. Multiple choice questions would be answered by clicking on radio-style buttons. Handling the programming portion would require students to create one or more program files and then upload them for grading. The upload process would also be done using the web interface. Once uploaded, the plan was for the programs to be automatically graded - a feature that is somewhat complicated. A student, Jacob Ratkiewicz, was hired to create the entire testing environment. Funding for this project was used to pay him on an hourly basis.

3. Test Implementation and Data Evaluation

Dr. Wolfer used his upper-level Operating Systems class as the first test-case for this assessment tool. This class was selected, with department approval, for testing since it is a capstone class attended primarily by senior computer science students approaching graduation. The test was administered during one two-hour course meeting period near the end of the semester, and it was held in a Computer Science laboratory. The actual testing mechanisms ran smoothly as students were able to access the test, fill in answers and upload program files all without incident. For this version of the test, automatic grading of the uploaded files was not implemented. Since the test was given, work has continued toward meeting this objective. More information regarding this feature will be provided in a subsequent section.

One aspect of the assessment test that was of concern was whether students would take the test seriously and devote proper effort to doing well. Since the test was not a requirement, it was a legitimate concern. Dr. Wolfer and Jacob monitored the test and it is their opinion that the students put forth appropriate effort. In this first sample 17 students took the exam, and the results were very interesting. Overall, the performance on the multiple choice questions, as gauged by the percentage of correct answers, was surprisingly low. Additionally, the performance on the programming portion was lower than expected. The next sections provide analysis of these results.

3.1 Suitability of Test Questions

A proper analysis of this data is complicated as there are many factors to consider. First, the questions themselves must be considered. Were they fair or appropriate? Was the program that was required fair and sufficiently doable in the time allotted? After much discussion, we have concluded that at this point we cannot answer these questions. We have determined that these initial results have established a baseline that will be useful for comparison when the test is given in the future. It is our opinion that the questions were fair and were of a nature that students at this level should have been able to score at a much higher level. However, our opinion can be confirmed or refuted only by having additional samples to consider.

3.2 Testing Specifics

Other factors to consider center on the specifics of the testing environment. For this first time, the test was given in a two-hour block near the end of the semester. Furthermore, the students in this class had just completed work on a course project that consumed a great amount of time and energy. Many of them indicated that they had been working late the previous night and consequently were taking this test with little or no sleep. It is conceivable that the students were simply too exhausted to do their best work, Therefore, in the future we will give the assessment test at a time that would work better for students. While the two-hour time limit seemed to be fine, it might be better if more time was allotted. Also, for the programming portion we think that some changes should be made. For example, a shell of the program that the students are to write should be provided. Hence, they stand a better chance of writing a proper solution, and what they create will be more easily handled by the automatic grading feature of the test.

4. Future Work

As mentioned, the first administration of the assessment instrument is useful in establishing a baseline of our upper-division undergraduate students' performance. We plan on giving this test again this coming fall but will enlarge the sample size. We will determine what course or courses to use as test cases and will schedule the exam near the end of the term, but not as late as was done this time. For this next testing cycle we plan on giving the same test and will compare the results to the first set of data. From that analysis we will decide on possible changes to the questions.

This assessment tool was created with a secondary purpose of use as a means to identify deficiencies in incoming graduate students. While still in its infancy, the graduate program has shown that it attracts many students who do not have a degree in computer science. Hence, it is important to identify what undergraduate courses these students need to take to prepare them for their graduate studies. It is hoped that this assessment test will provide useful data for consideration when these students are advised. Therefore, we plan on administering this exam to new graduate students in the near future. A side benefit of giving the test to this group of students will be that it will provide another set of data to use when determining the appropriateness and quality of the questions themselves.

Another area where work continues is on the automated program grading feature. Automatically grading multiple-choice questions is straight-forward. However, performing the steps necessary to properly grade a computer program is not. To accomplish this there must be a degree of uniformity maintained by the students when writing their programs. Additionally, a grading program, or script, must be written that will accept as input student programs and will generate output that can be compared to correct output. We foresee this as a process that will take additional effort and refinement.

Lastly, we foresee giving this test on a regular basis. Hence, there will be a need to continually evaluate the questions and update and/or change them. We hope to create a large pool of suitable questions from a variety of sources. Since we also have questions based on the department's electives, we hope to have our colleagues contribute to this project by forwarding us questions based on these courses.

5. Budget

The IU South Bend Assessment committee awarded us $640 to develop the testing software. This money has been used to pay our student assistant on an hourly basis. We have asked him to continue working on this project until the end of June as we still have funds to pay him. At the end of June whatever money that remains will be returned to the assessment committee.

6. Conclusions

Overall we are quite satisfied with what has been accomplished with this grant. An assessment instrument has been created - one that is highly flexible, easy to update and easy to administer. The real value of the test will occur when it has been given to subsequent groups of students. At that point we will be able to make more informed conclusions about the level of competence of our students. In turn, this information can be used to help the department assess the effectiveness of our courses and teaching. As the test is regularly given we also will be able to gauge the quality of the test questions making appropriate changes. By developing this web-based testing mechanism we have created a tool that will be quite valuable to the department for years to come.

7. Acknowledgment

We would like to thank the IU South Bend assessment committee for funding this project (2000/01 funding cycle).

Back to top of page