|
|
|
ChemCollective DLExamples |
The ChemCollective Digital Library3. Assessment
3.a. Learning assessmentAssessment of usability aspects of the virtual lab has been ongoing since the earliest versions. It takes about 5 minutes for a student to become sufficiently familiar with the software that their attention shifts almost exclusively to the chemistry goals. Our current instructors find the lab easy to implement in their classrooms, although this may reflect some bias in the sample of instructors who have used the lab and respond to our surveys. We also have student surveys from multiple sites that indicate students find the lab to be useful for their learning (typically 3.5 on a scale of 5), but feel the need for more direction while using the laboratory (also typically 3.5 on a scale of 5). Throughout the development we have also routinely observed student problem solving (mostly at Carnegie Mellon, but also with students at Florida Atlantic University, University of British Columbia and Pittsburgh’s Central Catholic High School). Robert Belford has also reported back extensively on student experiences at University of Arkansas at Little Rock. Classroom observations of students performing online experiments led to a particularly interesting finding. Initially, our instructional goal for these online experiments was to embed the procedural knowledge of the course in a context that highlights its utility, such that students learn not only how to do a procedure but also when to do it. Our observations, however, shifted our instructional perspective to one of helping students move beyond shallow problem solving strategies. In particular, we noticed that students employ a potentially superficial strategy in which they analyze the language in a word problem to discover the given and requested quantities and then search for equations that connect the given to the requested quantitities. For instance, a calorimetry text problem may give a measured change in temperature (ΔT given) and ask for a heat (q requested). A student, having first written down the ΔT and q from the problem statement will then sift through the equations in the current textbook chapter and identify q = m*Cp*ΔT as a connecting equation. While this is a potentially useful skill, it does not give students practice with the fundamental concepts underlying calorimetry. An activity that requires deeper reflection is our virtual lab that requires students to design an experiment to measure the enthalpy of a reaction. The text-matching strategy fails in experimental design and the student must instead realize that the equation q = m*Cp*ΔT represents an experiment in which a temperature change is used to measure heat. Our observations show that students find the experimental design problem considerably more difficult than the text problem, and often have as a first question "what equation do I use?". These observations suggest that online experiments promote additional conceptual learning. A more formal assessment effort occurred in the Spring 2004 semester at Carnegie Mellon, in which we collected extensive data regarding the effects of the use of Virtual Lab and scenario-based learning activities on students’ understanding of basic chemistry concepts. The semester was divided into three 5-week segments, corresponding to the course hour exams. Each segment had used situated activities: (i) online materials used only during recitation periods (1 hour time limit on activities, with help from human tutors), (ii) traditional paper-and-pencil homework activities and (iii) online homework (no time limit, with access to human tutors through office hours and email). All work handed in by students was saved, including recitation sheets, homework assignments, quizzes and exams. Student attitude surveys were given before each of the three exams, and included items that measure student views of the activities and their confidence with the various course concepts. Unannounced pre-tests were given in recitation the week before scheduled exams in the second and third sessions to assess students' understanding of concepts. These reflect the learning that took place through the activities prior to studying for the exam. The virtual lab was also instrumented to collect a trace of student interaction, and this data is available for the entire course. The principle findings are7:
Most recently, we conducted a well-controlled study of students enrolled in an online course (http://www.cmu.edu/oli/courses/chemistry/)8, 19 that uses ChemCollective materials. The results indicate that students who engage in virtual lab activities show a solid and direct relationship between the number of events of engagement with the virtual lab and learning as measured on a post test and controlling for variances in SAT scores. 3.b. Collection assessmentWe have chosen not to require users to identify themselves to gain access to the materials. While this may maximize use of the materials, it has the down side of making it challenging to monitor who is using the collection and how. To better understand our dissemination, we developed a strategy of attempting to estimate three factors: exposure to, use of and participation in the community. The strategy and results, up to 2005, are contained in a report entitled "The ChemCollective: Monitoring the Path from Seeing to Using to Contributing"20. The goal of the analysis was to estimate what percent of our target audience has been exposed to the ChemCollective materials (see), how many use those materials (use), and how many contribute materials (contribute). The results of the report, with updated numbers, are briefly summarized here. Target Audience: We consider the primary target audience of ChemCollective materials to be the teachers of introductory chemistry. Based on information from the U.S. Department of labor, we estimated 100,000 high school chemistry teachers and 9,000 college introductory chemistry instructors. See: Our estimate for exposure to the materials is based on number of CDROM's distributed at conferences, attendees at workshops and talks, readership of magazines and journals that have featured our materials, and number of unique visitors coming from search engines such as google and yahoo. This leads to an estimate of 7000 chemical educators who have been made aware of the materials. Use: Since the collection does not require users to register or log in, and the software can either be run from the web site, downloaded to the local computer, or run from a CD-ROM, we do not have complete usage statistics. From available statistics, we can however make a conservative estimate that 200 classrooms currently make extensive use of our materials. Contribute: Eleven instructors have contributed a total of 56 different activities to the collection. (These contributors work closely with our development team.) 8 groups have made contributed translations of the virtual lab interface along with over 70 activities (Spanish, Portuguese, French, Catalan, Galician, German, Russian, and Lithuanian) with 3 more languages in progress (Traditional Chinese, Polish and Check). We therefore have 22 current contributors. The web page invites instructors to contribute feedback on the use of the materials in their classroom and to participate in assessment studies. In the past four years, 40 instructors have provided feedback (although much of it is general and does not address specific educational issues or goals) and about 13 expressed interest in participating in studies. The above conservative estimates for our target audience are that 7000 instructors have seen the collection, 200 use activities in their classroom, 22 have contributed activities, and 40 have contributed feedback. These results give insight into the number of users that go from seeing to using a collection (3%), from using to contributing activities (11%), and from using to contributing feedback (20%). The following table is a brief overview of our web statistics:
| |||||||||||||||||||||||||||||||
| Last Updated: Sunday, March 30, 2008 @ 05:23:09 pm |
