1	"A New Approach to Performing..." A New Approach to Performing Course Evaluations: Using Q Methodology to Better Understand Student Attitudes Joe Jurczyk Susan Ramlo University of Akron Paper Presented at 20^th Annual Q Conference
2	Student Course Evaluations Conducted to measure the effectiveness and quality of: teaching course design other learning resources that are implemented in the classroom.
3	Student Course Evaluations By 1993 nearly 90% of all college institutions required student course evaluations (Seldin, 1993). “When student evaluations of faculty are used summatively to determine retention, promotion, and merit pay, there is the potential for serious consequences in the classroom.” (Adams, 1997)
4	Traditional Surveys Likert scale Open-ended questions Analyzed with summative data (e.g. average satisfaction with textbook: 4.4)
5	Purpose of Study Use Q to gain insight into student point of view. Understand groupings that exist in classroom Exploratory research to determine method feasibility, lessons learned, obstacles Research conducted at a two-year college in large, midwestern university
6	The courses evaluated 4 courses evaluated using Q methodology during lecture during the last week(s) of the course. Spring (1) & 8 week-summer session (3) 2004 All science courses within the same department within a Community & Technical college Each consisted of lecture & laboratory 2 Basic Chemistry (summer) 1 Technical Physics: Mechanics II (spring) 1 Technical Physics: Heat & Light (summer) 4 different lecture instructors
7	Basic Chemistry – Summer 04 Service course for various associate degree programs (criminal justice technology, fire protection, allied health) & pre-nursing students Summer sections were primarily female (gender not requested on questionnaire) Focuses on concepts of inorganic chemistry (minimum math) 2 different instructors Tenure-track teaching M & W, day time, 8 week session; taught both lab & lecture. Part-time instructor, Saturdays, 8 week session; taught both lab & lecture
8	Technical Physics 4 half-semester offerings (8 week sessions) Required, non-calculus physics courses for Engineering Technology programs Only 1 female student in each of the 2 courses evaluated (gender not requested on questionnaire) Technical Physics: Mechanics II Spring 2004, weeks 9 through 16 Tenured professor taught lecture Part-time instructor taught laboratory Technical Physics: Heat & Light Summer 2004, 8 week session Same part-time instructor taught both lab & lecture
9	Methodology: Q-sort materials 40 individual pieces of paper with statements regarding: Overall course structure Lecture quality Lab quality Lecture instructor Lab instructor quality Packet with instructions, questionnaire, & fill-in grid Grid for placing / arranging statements during the sorting process Voluntary & anonymous
10	Sample Statements 1. I had adequate time to complete lab exercises. 2. My instructor used teaching methods well suited to the course. 3. My instructor organized this course well. 4. My lab instructor was available during office hours. 5. Course assignments were interesting and stimulating. 6. Course assignments helped in learning the subject matter. 7. My lab instructor provided sufficient help in the lab. 8. My instructor was well prepared for class meetings. 9. The objectives for the lab activities were well defined. 10. I kept up with the studying and work for this course. 11. Lab facilities were adequate. 12. I actively participated in class activities and discussions. 13. My lab instructor was prepared for lab lectures and discussions. 14. I was interested in the content of this course before taking it. 15. My instructor adapted to student abilities, needs, and interests. Statements obtained from Indiana University Evaluation Manual: http://www.indiana.edu/~best/pdf_docs/multi-op_manual.pdf
11	Observations of the Q sort process Spring-semester 2004 – Mechanics II Per student comments - Students were frustrated because they had completed the required college, Likert- scale evaluation during the prior course meeting (redundancy). Per observations - Students were frustrated because of the lack of space available during the sort (tablet-desks). Students made negative comments regarding the amount of time required to perform the sort. Frustration led some students to stop the Q sort process &, therefore, negatively affected participation.
12	Observations of the Q sort process Summer 2004 – 3 evaluations Students did not evaluate summer courses with the college, Likert- scale instrument – may have lessened frustration. Students sat at or had available long tables – sufficient space to distribute the Q sort materials – seemed to lessen frustration. Nearly all students completed the survey in less than 20 minutes (less time than the spring Q sort). Some students had trouble distributing the 40 statements into 3 preliminary groups (e.g. liked instructor/course so all “agree”). Problems with preliminary sort seemed to prolong the Q sort process (last to hand in their sorts &, possibly, may have been among those who did not complete the sorting process).
13	Unsolicited student comments Complaints that the Q sort took longer to complete than the college’s Likert-instrument Statements that concerning difficulty discerning a difference between certain statements… 1 student gave the following example: 34. My instructor makes difficult material easily understandable 28. My instructor explains the material clearly.
14	Technical Physics - Evaluation 1 Factor Loadings Evaluations 1 2 3 1 0.3719 0.2048 0.0997 2 0.2457 0.0668 0.5204X 3 0.0013 0.9559X -0.0329 4 0.3428 0.3043 -0.0896 5 0.4727X 0.2248 -0.1832 6 0.4877X 0.1918 0.2056 7 0.5535X 0.0386 -0.3863 8 0.8290X 0.1916 0.0844 9 0.3149X 0.0260 -0.2403 % expl.Var. 21 13 6
15	Technical Physics - Evaluation 1 No. Statement No. Z-SCORES Factor 1: Self-confident 17 I feel that I performed up to my potential in this course. 17 1.802 18 The total amount of material covered in the course was reaso 18 1.698 10 I kept up with the studying and work for this course. 10 1.417 6 Course assignments helped in learning the subject matter. 6 -1.639 5 Course assignments were interesting and stimulating. 5 -1.763 20 Overall, I would rate the textbook/readings as excellent. 20 -1.912 Factor 2: Negative self 21 I knew what was expected of me in this course. 21 2.145 15 My instructor adapted to student abilities, needs, and inter 15 1.716 1 I had adequate time to complete lab exercises. 1 1.716 10 I kept up with the studying and work for this course. 10 -1.716 17 I feel that I performed up to my potential in this course. 17 -1.716 11 Lab facilities were adequate. 11 -2.145 Factor 3: Mixed feelings about lab & content 30 My lab instructor clearly explained the procedures to be use 30 2.145 29 My lab instructor promptly returned reports and assignments. 29 1.716 31 I learned a lot in this course. 31 1.716 14 I was interested in the content of this course before taking 14 -1.716 19 The labs were important to learning in this course. 19 -1.716 13 My lab instructor was prepared for lab lectures and discussi 13 -2.145
16	Basic Chemistry(weekdays)– Evaluation 2 Factor Loadings Evaluations 1 2 3 1 0.7810X -0.2394 0.3325 2 0.4703X 0.1917 0.0780 3 0.2884 -0.2943 -0.0957 4 0.1038 0.0912 0.5214X 5 0.4070 0.5233X -0.2594 6 0.5692X -0.0245 -0.3399 7 0.4692X -0.3349 0.0836 8 -0.3499X -0.0786 0.3212 9 0.6954X -0.4902 -0.0316 % expl.Var. 25 9 8
17	Basic Chemistry(weekdays)–Evaluation 2 No. Statement No. Z-SCORES Factor 1: Positive view of lecture instructor 34 My instructor made difficult material easily understandable. 34 2.170 8 My instructor was well prepared for class meetings. 8 1.403 35 My instructor answered questions carefully and completely. 35 1.227 5 Course assignments were interesting and stimulating. 5 -1.673 14 I was interested in the content of this course before taking 14 -1.718 32 Lab assignments were interesting and stimulating. 32 -2.125 Factor 2: Positive view of lab instructor 7 My lab instructor provided sufficient help in the lab. 7 2.145 1 I had adequate time to complete lab exercises. 1 1.716 13 My lab instructor was prepared for lab lectures and discussi 13 1.716 18 The total amount of material covered in the course was reaso 18 -1.716 14 I was interested in the content of this course before taking 14 -1.716 37 The course improved my understanding of concepts in this fie 37 -2.145 Factor 3: Well prepared for the course but it was not rigorous enough 14 I was interested in the content of this course before taking 14 2.145 10 I kept up with the studying and work for this course. 10 1.716 19 The labs were important to learning in this course. 19 1.716 17 I feel that I performed up to my potential in this course. 17 -1.716 31 I learned a lot in this course. 31 -1.716 18 The total amount of material covered in the course was reaso 18 -2.145
18	Basic Chemistry(Saturdays)–Evaluation 3 Factor Loadings Evaluations 1 2 3 1 0.1718 -0.0627 0.4974X 2 -0.0343 0.1172 0.1148 3 0.0048 0.3192 0.6895X 4 0.6831X 0.2150 -0.0145 5 0.3615 0.5877X -0.1597 6 0.3028 -0.1192 0.4756X 7 0.0048 0.6455X 0.1408 8 0.8766X 0.2399 -0.0502 9 0.2853 0.7446X -0.0565 10 0.2847 0.1141 -0.2582 11 0.6251X 0.2099 -0.0381 12 -0.0464 0.1142 0.3583X 13 -0.1330 0.1896 0.4545X % expl.Var. 16 13 11
19	Basic Chemistry(Saturdays) – Evaluation 3 No. Statement No. Z-SCORES Factor 1: Positive self / negative instructor 17 I feel that I performde up to my potential in this course. 17 2.175 10 I kept up with the studying and work for this course. 10 1.709 12 I actively participated in class activities and discussions. 12 1.689 36 This course increased my interest in the subject matter. 36 -1.552 28 My instructor explained the material clearly. 28 -1.630 35 My instructor answered questions carefully and completely. 35 -2.097 Factor 2: Hands-on/ negative instructor 11 Lab facilities were adequate. 11 1.970 12 I actively participated in class activities and discussions. 12 1.449 1 I had adequate time to complete lab exercises. 1 1.353 35 My instructor answered questions carefully and completely. 35 -1.940 34 My instructor made difficult material easily understandable. 34 -1.988 28 My instructor explained the material clearly. 28 -2.336 Factor 3: Positive instructor / negative course structure 2 My instructor used teaching methods well suited to the cours 2 1.647 33 My instructor showed genuine interest in students. 33 1.406 8 My instructor was well prepared for class meetings. 8 1.352 32 Lab assignments were interesting and stimulating. 32 -1.866 20 Overall, I would rate the textbook/readings as excellent. 20 -2.008 5 Course assignments were interesting and stimulating. 5 -2.036
20	Summer Heat and Light – Evaluation 4 Factor Loadings Evaluations 1 2 3 1 -0.0707 -0.4054X -0.1373 2 0.6646X 0.0164 0.1522 3 0.6470X -0.3430 0.0119 4 0.3853 0.6209X -0.1026 5 0.7511X 0.3355 -0.1252 6 0.5358X -0.2429 -0.0246 7 0.4503 -0.3274 -0.3839 8 0.6258X -0.1290 -0.2567 9 0.5801X 0.2454 0.2367 10 0.5242X -0.0636 0.0956 11 0.6849X 0.0194 -0.0606 12 -0.1215 -0.4938 0.5432X % expl.Var. 30 11 5
21	Summer Heat and Light – Evaluation 4 No. Statement No. Z-SCORES Factor 1 Positive instructor / negative materials 29 My lab instructor promptly returned reports and assignments. 29 1.967 34 My instructor made difficult material easily understandable. 34 1.767 2 My instructor used teaching methods well suited to the cours 2 1.374 20 Overall, I would rate the textbook/readings as excellent. 20 -1.641 32 Lab assignments were interesting and stimulating. 32 -1.662 11 Lab facilities were adequate. 11 -2.414 Factor 2 – Positive self – negative lab & materials 31 I learned a lot in this course. 31 1.779 17 I feel that I performed up to my potential in this course. 17 1.765 36 This course increased my interest in the subject matter. 36 1.765 16 Lab sessions were well organized. 16 -1.594 20 Overall, I would rate the textbook/readings as excellent. 20 -1.765 24 My lab instructor related lab exercises to lectures and read 24 -2.121 Factor 3 - Good overall instruction / negative self 8 My instructor was well prepared for class meetings. 8 2.145 21 I knew what was expected of me in this course. 21 1.716 24 My lab instructor related lab exercises to lectures and read 24 1.716 1 I had adequate time to complete lab exercises. 1 -1.716 17 I feel that I performed up to my potential in this course. 17 -1.716 10 I kept up with the studying and work for this course. 10 -2.145
22	Recommendations Single facilitator – provides consistent instructions to students and ability to compare observations of multiple groups (classes) Findings improve with larger classes – more stable factors Future research: multiple evaluations - study changes over time: between individuals and between groups
23	Questions ? Joe Jurczyk – jurczyk@apk.net Sue Ramlo – sramlo@uakron.edu Paper and Presentation available at: http://www.projectsbyjoe.com/issss2004