General Education Assessment 1997-98:

Critical Thinking Assessment Battery (CTAB)

Date: February 22, 1999

Office of Assessment

Winthrop University

Rock Hill, South Carolina 29733

INTRODUCTION

The present report summarizes the procedures employed in the Critical Thinking Assessment Battery pilot project sponsored by American College Testing (ACT). Winthrop results, conclusions and implications are also presented.

INSTRUMENTATION and PROCEDURES

Description of the CTAB Instrument

The CTAB instrument involved three parts: essays written after audio prompts, objective test items to measure critical and applied reasoning, and documentation of self-directed learning activities. Writing was assessed for the following: audience, organization and language. Critical thinking in the writing samples was assessed for students’ ability to articulate the benefits, problems and alternatives associated with posed arts and humanities, social science, and physical science contextual situations.

Data collection procedures

Winthrop faculty from all academic units were invited to participate in the CTAB pilot study. Cooperating faculty selected courses in which the CTAB would be administered. Student participation was voluntary. All students enrolled in the selected courses were given a written explanation of the CTAB pilot project and asked to sign informed consent agreements before participating. The CTAB was administered to Winthrop University students during the Fall, 1997 term by Assessment Office faculty and staff trained in standardized procedures to administer the pilot instrument. In order to accommodate as many students as possible, the CTAB was administered during regularly scheduled class times and during evening hours.

Demographic data collected

Student data were collected for the following: (1) college/major (2) gender, (3) race/ethnicity, (4) age (5) SAT or ACT college entrance examination score, (6) grade point ratio (GPR), (7) grade level and (8) status as either a full or part time student.

Data analysis procedures

Winthrop faculty volunteers and Office of Assessment staff were trained to assess writing samples using ACT’s rubric. CTAB materials were then returned to ACT for analysis of objective items in Part Two of the CTAB pilot instrument.

RESULTS

Winthrop received ACT’s final CTAB Summary Report on November 13, 1998. ACT returned the following data: individual Winthrop student scores for persuasive writing (PW), critical thinking (CT) and applied reasoning (AR) and a comparison of Winthrop scores with mean scores for the national ACT pilot sample.

Winthrop respondents (n = 202) represented the following college affiliations: (1) Arts and Sciences (n = 83, 41%), (2) Business Administration (n = 52, 26%), (3) Education (n = 57, 28%), Visual and Performing Arts (n = 8, 1%). More than 90% of respondents (n = 183, 92%) reported attending Winthrop as full time students with 155 students (n = 79%) reported spending > 13 hours per week in class.

Seventy-seven percent of Winthrop respondents (n = 156) were females compared to 58% (n = 1,347) of the pilot sample (n = 2,177). Fifteen percent (n = 319) of the pilot sample and 22% (n = 45) of the Winthrop sample were African-American. Seventy-nine percent of the pilot sample and 73% of the Winthrop sample were Caucasian. Most Winthrop participants were aged 19-24 (82%) and 9% (n = 18) were aged 30 or more. ACT did not report frequency of responses by students’ ages.

Correlation between SAT or ACT scores and Grade Point Ratio (GPR)

Pearson’s Product Moment Correlation was used to evaluate the relationship of SAT verbal and math scores to Grade Point Ratio (GPR). SAT verbal and math scores were moderately correlated to GPR (r = 0.40, p = 0.0001 and r = .41, p = 0.0001, respectively). The correlation of ACT scores to GPR was r = .32 , p = 0.1386 (ns). The absence of significance may be a function of the relatively small number of students for whom ACT scores were reported (n = 23). No significant difference in GPR by student class rank was identified (F = 2.21, p = 0.0694).

Part One: Writing samples

Writing samples were obtained from 131 Winthrop seniors (65%) , 35 freshmen (17%), 24 (12%) sophomores, 10 (5%) juniors and two graduate students (1%) . Freshmen and Seniors were the target student population for the national pilot.

Winthrop faculty scored Winthrop student writing samples. ACT also reported persuasive writing scores for Winthrop University (WU) students. The mean persuasive writing (PW) score for Winthrop students was 24.3 (ACT four year institutions mean PW score was 22.3 with a possible score range from one through 36).

Part Two: Critical thinking and applied reasoning

Mean critical thinking (CT) scores for Winthrop students was 21.6 (ACT four year institutions mean CT score was 20.1 with a possible score range from one through 32). ACT reported mean subscores for Applied Reasoning (AR) (possible range, one through 32) for Winthrop were slightly higher than mean subscores for all institutions participating in the ACT pilot. Specific subscores for Applied Reasoning were as follows:

Part Three: Engagement Activities

ACT reported that data collection problems precluded issuance of summary data on Part Three.

Attitudinal data

ACT separated the pilot sample results by whether respondents attended two- or four-year colleges. Winthrop data are compared to the ACT four-year pilot sample results.

Responsibilities and Time Allocations

Students were asked to report the approximate number of hours they spent each week in the following activities: (1) learning activities outside of class time, (2) paid work (employment) of any kind, (3) caring for family and others, (4) volunteer work for civic, political or religious organizations, (5) television viewing for leisure, (6) computer activities, including Internet, for personal interest or pleasure, and (7) sports and physical exercise.

Half of the Winthrop students and about 45% of ACT respondents reported spending one to ten hours a week in learning activities outside of class time. Approximately 42% of Winthrop respondents (and 43% of the ACT sample) reported spending 11-25 hours a week in learning activities outside of class time.

Thirty-one percent of the ACT sample and 28% of the Winthrop sample reported that they were not working for pay. Almost half of Winthrop respondents (46%) reported working part-time (one to 20 hours per week). More than half of Winthrop students reported spending one to 20 hours per week caring for family members and others. Three-fourths of the Winthrop sample reported that they spend one to 15 hours a week volunteering for civic, political or religious organizations. Seventy-eight percent (n = 169) of Winthrop respondents said they spend between one and ten hours a week on sports and physical exercise. No ACT data were reported for the amount of time respondents work part-time, care for others, volunteer for organizations or exercise.

Seventy-five percent of both Winthrop and ACT CTAB participants reported that they watch one to ten hours of leisure television each week. Sixty percent (n = 121) of Winthrop respondents and 57% of ACT respondents noted that they spend one to five hours per week using a computer for personal interest or pleasure.

Attitudes about the importance of educational activities and courses

Students were asked to report the relative importance of a variety of courses which they have either taken or plan to take, based upon their career plans. Using a Likert-type scale, students were asked to rank the following types of courses:

(1) General Education courses

(2) Courses which emphasize applying knowledge outside the classroom

(3) Courses which emphasize values and political and social aspects of issues

(4) Courses which emphasize investigating and solving problems

(5) Courses which emphasize the development of communication skills

More than half of Winthrop respondents (n = 114, 56%) and 55% of ACT respondents said that general education courses were either extremely valuable or relevant and valuable. Only 7% (n = 14) of Winthrop respondents and 9% (n = 195) of ACT respondents said that general education courses had little or no value to the respondent personally.

Eighty percent of ACT respondents (n = 1,729) and 82% of Winthrop respondents (n = 165) said that courses which emphasize applying knowledge outside the classroom were very valuable or relevant and valuable. Only 1% (n = 2) of the Winthrop sample and 4% (n = 77) of the ACT sample said that courses which emphasize applying knowledge outside the classroom were of little or no value.

Courses which emphasize values and political and social aspects of issues were viewed as very valuable or relevant and valuable by more than half of Winthrop respondents (n = 113, 56%) and ACT respondents (n = 1133, 52%). Only 6% (n = 12) of Winthrop respondents and 12 % (n = 249) of ACT respondents reported that such courses had little or no value to them personally.

All students reported that courses which emphasize investigating and solving problems had some value to them personally. More than 80% (n = 164) reported that courses which emphasize investigating and solving problems are very valuable or relevant and valuable. Only 2% (n = 5) reported that such courses have little value to them personally. More than 90% (n = 182) of respondents said that courses which emphasize the development of communication skills were either very valuable or relevant and valuable. All students reported that such courses had some value to them personally.

Amount of emphasis given to growth in the developmental foci: intellectual, personal, social, and preparation for careers and/or further study

Students were asked "At this point in your life and your college education, indicate the amount of emphasis you are giving to each of the following: (1) intellectual growth (acquiring knowledge, skills, ideas, concepts and analytic thinking), (2) personal growth (developing self-understanding, self-discipline, mature attitudes and values), (3) social growth (understanding others and their views, adapting to a variety of social situations) and (4) preparation for a career and/or further study. Students were asked to select one of the following three ranks: (1) little emphasis, (2) moderate emphasis or (3) great emphasis) for each of the four items. The highest percentage of Winthrop respondents reported giving "great emphasis" to preparation for a career and/or further study (84%) followed by personal growth (75%), intellectual growth (72%) and social growth (64%).

DISCUSSION

There were clear limitations in the project due to the small Winthrop student sample and the experimental nature of the instrument and associated procedures. Nonetheless, results of the present critical thinking assessment initiative are consistent with those obtained on previously administered, nationally standardized general education examinations in suggesting Winthrop student performance at or slightly above the national average. Results also suggest that the activities and attitudes of Winthrop participants relevant to general education and related areas are generally consistent with their scores nationally.

Positive features of the CTAB

The CTAB addressed critical thinking in the humanities and the physical and social sciences. Multiple assessment strategies were used including: (1) audio-prompted writing samples from the social and physical sciences, and the arts, and (2) objective items to measure applied reasoning and critical thinking. The CTAB also incorporated cultural diversity in the audio prompts used for the writing samples.

Problems in using the CTAB for General Education Assessment

The time frames for individual portions of the CTAB were not consistent with typical class periods (50 or 75 minute classes), making in-class administration of the instrument difficult. In addition, Winthrop faculty raters viewed the rubrics for assessing writing samples of the three audio prompts to be incomplete, ambiguous, and to demonstrate gender bias. Volunteer respondents and raters perceived the quality of audio prompts used to direct the writing samples to be poor. Winthrop University had standard institutional-style audio equipment for classroom use. During ACT workshop training sessions in Chicago and Atlanta, the problem of poor audio tapes was identified by multiple participating institutions.

Participation of Winthrop faculty and students

Favorable levels of student participation despite short timelines were likely the result of faculty support and willingness to incorporate the general education assessment project into regularly scheduled class time. Faculty motivation to participate in the CTAB general education assessment pilot also seemed to be important to student motivation although this relationship was not measured. Assessment Office faculty believe that general education assessment experiences obtained during students’ college careers can help students to perform future self-assessment of life-long learning needs.

Other positive aspects of the CTAB project

The in-class testing environment ( using standardized audio prompts, scripted written and oral instructions and defined time constraints) helped to control for extraneous factors which influence assessment initiatives. Conducting the general education assessment activity primarily during regularly scheduled class time also facilitated obtaining the larger sample of seniors.

Using the CTAB pilot instrument to perform general education assessment during the 1997-98 academic year also enabled Winthrop to assess individual students rather than an aggregate of freshmen or seniors. Using the dataset established through the CTAB general education assessment project in 1997-98, individual students can be tracked longitudinally during their educational experience at Winthrop University. Longitudinal data can help demonstrate the benefits of educational experiences

Future plans

Winthrop’s participation in the CTAB pilot affirmed the position of Assessment Office faculty that in order for general education assessment to be more effective, the assessment process, in particular for seniors, needs to be standardized and, in some manner, made mandatory. ACT reported that pilot results did not warrant pursuing the CTAB project further at the present time. Winthrop will probably return to using the Institutional Matrix form of the College BASE during the 1998-99 academic term while the General Education Program is being re-evaluated and revised. However, the Assessment Office will attempt to secure the cooperation of Winthrop administrative and academic units to make the general education assessment process mandatory for seniors. Additionally, with the assistance of the General Education Committee and the Faculty Advisory Board on Assessment and Institutional Effectiveness, the selection of instruments and procedures most relevant and practical for assessing general education outcomes, including those pertaining to critical thinking/problem-solving will be examined on an ongoing basis in the future.

REFERENCE LIST

     Cook P., Johnson R., Moore P., Myers P., Pauly S., Pendarvis F., Prus J. and Ulmer-Sottong L. (1996). Critical Thinking Assessment: Measuring a Moving Target: Report and Recommendations of the South Carolina Higher Education Assessment Network Critical Thinking Task Force, June.
     Lutz D. and Dendurent S. (Oct. 27, 1998). Final CTAB Summary Report. ACT Department of Postsecondary Services.