Measuring the impact of an information literacy session

. Abstract This research was undertaken at Sheffield Hallam University using formally assessed student assignments within a mandatory module in one of the four faculties in the university: the Faculty of Development and Society. It aimed to measure the impact of a teaching session on the quality, i.e. authority of references. The content of the session focused on how to efficiently find relevant and authoritative information for an academic essay by developing information literacy skills. Measuring the impact of information literacy teaching on student research behaviour and ultimately on student achievement is uncommon in practice and has methodological problems. If achievable such a measure would guide planning and delivery of further sessions and provide further evidence of effectiveness for teaching staff and central services management. The research looked at the development of student information literacy abilities, before and after a teaching intervention, as measured by the scholarly nature of references as a proportion of total references. The results indicated a significant improvement from a scholarly index of 0.25 or 25% in the assignment before the intervention to 0.76 or 76% in the one afterwards. There are still issues with the methodology as it does not isolate other factors such as other interventions and autonomous learning. However it does provide one indication of a positive outcome of the sessions and for further research.


Introduction
This paper describes a study to measure the impact of an information literacy session. Information literacy is defined by the Chartered Institute of Library and Information Professionals (2012): 'Information literacy is knowing when and why you need information, where to find it, and how to evaluate, use and communicate it in an ethical manner".
The session was delivered by Information Advisers, who are based in the Adsetts Learning Centre, a facility containing library and media resources, a range of study spaces, computers and printing services. Information advisers work within Student and Learning Services, a central services department, and are the equivalent of subject librarians at other universities. They work as part of a team supporting the Faculty of Development and Society through delivering and supporting Faculty teaching, ensuring that library resources are accessible relevant and up-to-date, updating the Faculty on central department news/services and gathering, then responding to feedback.
The teaching Information Advisers deliver is based on the Information Literacy Framework (Sheffield Hallam University, Learning and Information Services 2011), a strategy that has been tabled at Faculty committee meetings. This Framework draws upon the SCONUL's (Society of College, National and University Libraries) Seven Pillars model of information literacy. The pillars are explained in some detail by Webber (2007) on the SCONUL website. They are also described in diagram form below (SCONUL 2007) to show that there are bases to the pillars: basic library skills and basic IT abilities and to show a progression from novice to expert level across each pillar.
As part of the Framework our minimum offer to first-year students is an induction to services and resources followed by a workshop looking in more depth at information literacy skills. These usually include, depending on the needs of the students and curriculum, how to develop a search strategy, selecting and accessing information/data sources, searching for information/data, critically evaluating information and information sources as well as managing and using information in an ethical manner. In this context ethical manner includes citation/referencing and an awareness of plagiarism.

Case Study
Module: Studying Environments, first-year core module Academic year: 2009/10 Programme: Urban and Regional Studies Module leader: Jo Jenkins Information literacy sessions delivered by colleauges Frances Hyde and Caroline Fixter

Intervention
In the first semester these particular first-year students had a brief induction (20-minutes) to library resources and Adsetts Learning Centre services in a lecture situation. These sessions specifically dealt with:  How to analyse an assignment question to determine what information is needed.
This included identifying/understanding what the main concepts are and developing an effective range of keywords to use as search terms;  Selecting the information sources that best answer the question;  Constructing an effective search strategy, e.g., how to combine keywords using Boolean logic (if this is appropriate to the information source);  Searching a variety of appropriate resources;  Critically evaluating the results;  Using the information in a responsible and ethical way, e.g. through appropriate citations.
The sessions began with an information literacy skills questionnaire to check knowledge and use of information sources such as journals and information databases such as ScienceDirect.
It also asked questions about information searching techniques, both to provide the Information Advisers with data on the level of skills of students had and to act as a benchmark to measure subsequent improvements.
There was also an exercise asking students to identify what sort of research animal they were from a list of possible research behaviours, the most appropriate being an 'information owl'. Information owls use a wide variety of different resources which change depending on what they are doing, whereas a shark, for instance, indiscriminately scoops up anything that goes by even if it is not useful.
A brief PowerPoint outlined the major relevant information retrieval tools available such as LitSearch (the then host or gateway for online information databases: mostly providing access to journal articles, conference papers, data and regulatory materials).
There was a demonstration of LitSearch and using Boolean searching and other search techniques such as truncation and phrase searching. These were presented in the PowerPoint below:  Boolean logic • And connects search terms.
 sustainable and design • Phrase searching  "sustainable design" • Or looks for synonyms  environment or ecology • Truncation looks for the stem of a word and any letters following  sustainab* will find sustainable, sustainability Some time, usually 20-25 minutes, near the end of the session was given for students to research their topic and to begin an exercise that required them to find a variety of sources and to write a bibliography of those sources. As an example, part of the exercise asked them to find an academic journal article which is relevant to their studies. Students were asked to complete these after the session and in time for the following week. They were then marked and returned to students, though this mark did not count towards their overall module grade. Common issues with the exercise were addressed at the following week's session.
As Information Advisers often do not see the same students regularly, they have little direct knowledge of the impact of their sessions. We need this information together with session evaluations by students in order to ensure that the staff time we are using is effective. The team of Information Advisers supporting this Faculty delivered 538 sessions and over 700 hours of teaching in the 2010/11 academic year. Measuring the quality of sources referred to in assignments before and after an intervention seemed to offer a substantive measure of many of the skills an information literate student would have. These would include selecting and accessing information from a variety of scholarly sources and referencing them.
We were also working in an organisational environment where reviews of our services were taking place in order to rank them in terms of importance to the university. We were also being asked to provide measures of impact for a working group within our central services department.

Methodology
We looked at some of the literature on measuring impact of information literacy sessions using student's referencing/citations. One in particular from Northumbria University by Middleton (2005) seemed to be applicable to our sessions in that we were teaching an information literacy session in semester two after a group of students had already submitted one assignment that required a reference list in the first semester.
I was impressed by the paper's emphasis (Middleton 2005) on measuring the quality of referencing as well as the quantity. This was one of the few outputs that would logically measure the impact of sessions that emphasised strategies to find relevant scholarly information and data from reliable academic sources. Middleton was based at a UK university and the research conducted with a group of undergraduate students. The paper had been published in a peer-reviewed academic journal, which gave us some confidence in the quality of the research.
The practical steps needed to use the methodology were relatively simple to conduct, looked to be time efficient and seemed to give a measurable indication of the effects of information literacy teaching, though it was recognised that this would not completely isolate the impact of our sessions from self-learning and other interventions in the time between assignments.
We aimed to measure the quality of sources in reference lists in the module assignment before the information literacy intervention and again in the assignment that followed. For this the scholarly index (SI) as used by Middleton (2005) and described below was followed for each assignment. This measured the proportion of referenced sources in an individual or group of assignments that are judged to be scholarly, i.e., judged as being from an authoritative publication, such as a peer-reviewed journal. Other categories of scholarly material are listed below.
The scholarly index (SI) calculation was made as follows: number of references in scholarly categories* total number of references less number of unidentifiable references * Scholarly categories were classed as follows:  books (scholarly -check against Catalogue)  journals (scholarly -peer reviewed/professional)  non-scholarly journals (weeklies, trade journals/magazines)  scholarly websites (official, professional, educational e.g., ending with .edu, .ac, .gov,)  non-scholarly websites (all others)  other scholarly (conference papers/proceedings) For example, if all 20 referenced sources in a bibliography were judged to be scholarly the scholarly index (SI) would be 20 divided by 20 = 1. If half were scholarly the SI would be 10 divided by 20 = 0.5. These SI figures would equate to 100% and 50% respectively. Currency, or whether the source material reflects current thinking, when it was created, published and updated was also an optional measure of the quality of references, but was not used in this exercise due to staff time constraints. It is an option for further research where the currency of references is relevant to an assignment or research project.
We were not concerned about the accuracy of referencing except where that meant we could not follow up references. Where this occurred, we classified the reference as unidentifiable.
Other criteria were considered in the process of reference list checking. There were references to specific websites that might have been considered entirely appropriate to the assignment topic such as local government sites, but not within the definition of scholarly that we used. A scholarly source was considered one that had an authoritative author or institution/organisation from an academic or professional field, whose intended audience was professional/academic. Books were considered scholarly if they had met the above and/or were on the Library Catalogue. If a journal was peer-reviewed, or a recognised professional journal, it was considered scholarly. Websites who met the criteria of authority and intended audience were considered scholarly. Having their URLs ending with.ac, .edu, .gov gave us an indication. Personal judgement was made as to whether a source met the criteria, but this was consistently applied between students and across the two assignments.
The information literacy questionnaire mentioned above was intended to give us another measure to triangulate with the referencing authority/quality measure. Unfortunately time pressures meant that the planned follow-up audit that would have given us more measures of specific information literacy abilities was not followed-up. If the research was repeated this work followed by a similar exercise, asking different questions, but testing the same skills, would be used. For example, one of the questions was: If you entered the keyword search term Inclus* into the Library Catalogue (Classic or Encore), this is using the truncation symbol, which words would you expect the catalogue to retrieve or search for? Include as many terms as you can. To avoid this being exactly the same question and therefore to be able to measure learning, we would at least change the example word Inclus*. Also for consideration in a further study some questions that tested particular skills, rather than confidence or use-levels, would enable us to measure improvements in information literacy abilities more directly. For example, as part of a study to assess the information literacy skills of undergraduate students in Quebec universities (Mittermeyer and Quirion 2003, 39) used this question (amongst others): In order to find more documents on your topic you can include synonyms in your search statement. To combine those synonyms in your statement, you should use:

a) AND b) + c) NOT d) OR e) Other (please specify): f) Don't know
We had also planned focus groups with students to directly ask them about the impact they thought the sessions had had on their studies. Due to time pressures this did not happen.
Students filled out an agreement for research form to allow us to use their reference lists. We used those assignments that had received consent from their authors and were sent to us by tutors. The form made it clear that the results of the evaluation/research would not include their names and would not count towards any of their assessments. Therefore this was a self-selecting group of students, who may have consented to their work being studied in the belief that they had submitted a good piece of work, especially if their first assignment had received good marks. We were unaware of the mark each assignment had received.

In Practice
We used the reference lists from 13 students and used the table below to count scholarly and non-scholarly references. The scholarly index figure for each assignment was calculated for the group of students as a whole.

Results
First assignment before the information literacy session: 22 scholarly references 95 total references less 6 of unidentifiable references (89) Twenty-two divided by 89 gives an index score of 0.25, i.e., 25% or a quarter of references were scholarly.
After the session the reference lists from the same students for their second assignment were evaluated. Results as follows: 119 scholarly references 156 total number of references less 7 unidentifiable references One hundred and nineteen divided by 149 gives a scholarly index of 0.79, i.e., 79% or over three quarters of references are scholarly. The results indicated a marked increase in the quality and quantity of referencing.

Reflection
Judgement of a scholarly reference was time-consuming and complicated by many nonstandard references. There was also a difficult judgement to make where references used professional websites that were scholarly to the topic/subject area. However the same criteria were used for both assignments. It might have been useful to also have the tutor calculating the scholarly index of the reference lists to provide a balance against what we judged as scholarly.
The two assignments we evaluated were not asking for the same type of information materials, which may have impacted on the difference in the quantity and quality of their referencing we found.
Other factors may have skewed the results, such as other teaching interventions and/or a learning progression due to study time between the assignments.

Disadvantages
A lot of time was needed to research, plan and implement the sessions. There were delays due to need to get formal student agreement, which was not originally anticipated. Due to time constraints parts of the original plan were not followed through, for example, a focus group with students to discuss their reflections on the two assignments and the impact they felt the interventions had had.
The methodology gave a possible link to the impact of our sessions, but on its own could not isolate the sessions as the only factor behind improvements in referencing quality. There was no control group of students who did not receive the sessions to compare results with. This was considered initially, but rejected due to concerns about equity.

Benefits
We identified a workable measure of impact and identified a strong progression in the scholarly nature of referencing between assignments. The exercise fostered closer liaison with teachers in the Faculty, which lead to further development of the sessions for the following year including links to direct assessment of referencing skills. The process of conducting the research, new to us, was a learning experience and gave us some insight into research methodology.