As promised, this post reviews a recent (2012) article from Cell Biology Education: Life Sciences Education that describes the development and validation of a freely-available concept inventory that assesses students’ scientific inquiry skills. I am also quite excited to be able to provide a preview of the promising results we obtained from a trial of the instrument with a sample of 200 students enrolled in a first year general biomedical science course at an Australian university.
Gormally, C., Brickman, P., Lutz, M (2012) Developing a Test of Scientific Literacy Skills (TOSLS): Measuring Undergraduates’ Evaluation of Scientific Information and Arguments. CBE – Life Sciences Education 11 (4): 364-377.
Firstly, the authors label the concept inventory as a measure of “scientific literacy” skills, but ‘scientific reasoning’ would also be a fair label, and the specific skills described in the article overlap very well with the skills in our LTAS TLOs in Science TLO3: Inquiry and Problem Solving, and TLO1.1 which refers to an understanding of the link between the methods of scientific inquiry ad the contestable nature of scientific knowledge. Specifically, the inventory tests 9 skills that include identifying valid scientific arguments, evaluating the use and misuse of scientific information, reading and interpreting graphical representations of data, understanding and interpreting basic statistics, and justifying conclusions based on quantitative data.
The thing that most impressed me about the inventory was the extensive validation undertaken by the authors to ensure the content validity (how we know the instrument measures ALL of the aspects of the concept/skill it claims to measure) and construct validity (how we know the instrument actually measures what we think it measures). Not only does the approach convince the reader of the validity of the inventory, but the descriptions of these validation methods are also clear and easy to read. If you were thinking about developing any sort of concept inventory, then this article is worth reading from a methodological perspective – a daunting amount of work, but very inspiring
The major advantage of this inventory over similar instruments (and the authors provide a decent review of several of these instruments for biology concepts, and general critical thinking skills), is that this inventory consists of a series of 28 multiple-choice questions. This means the inventory can be used with very large cohorts with very quick turn around times. The inventory takes students 35 min to complete, so it can be administered within a lecture, tutorial or practical class setting. The inventory has also been designed, developed and validated with students in introductory biology courses, so it is particularly relevant to early stage students undertaking our biomedical science courses. Now here is the punch-line: the entire inventory and how to score student responses is available in the Supplementary Materials, it is free to use! The article does recommend contacting the authors before using it – I emailed both Cara Gormally and Peggy Brickman and received a very prompt and enthusiastic response. The inventory was designed to be used as either a diagnostic to help tailor curricula to your students’ needs, and/or as a pre-, post-test design to determine the impact of a new teaching intervention or curricular change. So, does it work on our context…?
We are addressing several questions in this initial trial:
1. Will the wording in any of the questions need to be adapted to the Australian context?
2. Will the inventory provide a reasonable level of discrimination in our first year cohort biomedical science cohort?
3. Is there a test-effect?
4. Will our inquiry-based practical curriculum result in changes in students’ thinking that are detectable by the instrument?
5. Will differences in students’ performance on the inventory be reflected differences in students’ performance in the assessment items associated with the practical curriculum?
Myself and the course practical coordinator completed the inventory and did not find any questions that contained USA-specific language or assumptions about USA geographical knowledge that would not translate well to the Australian context. We therefore used the inventory verbatim with our students.
We administered the inventory in the first practical class in our first year, first semester biomedical science course at UQ earlier this month. We run 7 repeat practical sessions for the 800+ cohort, on a fortnightly cycle, with each class accommodating a maximum of 112 students. Two classes were selected to represent the diversity of degrees present in the cohort (the majority of students are enrolled a range of degrees including Bachelor of Dentistry, Bachelor of Pharmacy, several sports-science degrees and science-based degrees). We gave the students the allotted 35 min and observed that very few students finished more than 3 min early, and 99% completed the first 20 questions, 85% made it to question 25 and 75% completed all 28 questions. In total, 212 students undertook the inventory, with only 2 students achieving the maximum score of 28/28, and the majority of students scoring 14/28 – 25/28. So far, this seems to indicate that the test is reasonably well suited to our first year students.
We will be using this instrument with these same students again at the end the semester, along with an additional ~200 students in two more practical sessions. This will allow us to tease apart the impacts of undertaking our inquiry-based practicals throughout the semester, and possible test-effects (ie students improving because they have taken the test previously). We will also be comparing student performance on the inventory with their performance in the practical assessment items to determine the level of consistency between these different assessment approaches. I look forward to releasing more of our results as the semester unfolds…
Happy practical class teaching
The trial reported here was conducted by Kirsten Zimbardi, Lesley Lluka, Prasad Chunduri and Judit Kibedi at the School of Biomedical Science, The University of Queensland. We are grateful to all of the students who participated in this trial.