- A Framework of Guidance for Building Good Digital Collections
Collections Principle 6
Collections Principle 6: A good collection has mechanisms for collecting data that measure use and usefulness.
Digital collections should be evaluated periodically to monitor usage, assess service effectiveness, demonstrate return on investment, inform collection development, inform strategic planning, and support funding requests. The criteria, methods, and metrics for evaluating collections will vary by the objectives of the collections and the purposes of the evaluation. For example, the collections of the National Science Digital Library are designed to support teaching and learning, so it is appropriate that evaluation measures focus on the educational impact of these collections.
Effective collection management employs a variety of research methods to assess collection usefulness. Observation, surveys, focus groups, interviews, experiments, case studies, and transaction log analyses have been used by digital libraries to assess usage and usability. Each method has its strengths and limitations. To obtain a clear picture of the value of a digital collection is to answer the question: “Who is using what, how, and why?” It is often necessary for collection evaluators to use a combination of methods and measures to answer this question effectively.
The use of the digital collection is closely related to the collection’s content, functionality, usability, and accessibility. Establishing benchmarks for use, collecting usage data over time, and following international standards for measuring use of digital content will enable collection managers to conduct longitudinal collection assessment and compare collection services with those provided by peers. Evaluation is an iterative process. Results of evaluation should inform the design and improvement of a digital collection.
Frameworks and guidance for evaluating digital collections:
- Fourth DELOS Workshop, Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics (2002) http://www.sztaki.hu/conferences/deval/presentations.html. Offers a promising evaluation scheme by identifying Users, Data/Collection, System/Technology, and Usage as four dimensions of digital libraries and developing evaluation metrics for each.
- Christine Borgman, Evaluating the Uses of Digital Libraries (2004) http://www.delos.info/files/pdf/events/2004_Ott_4/Borgman.pdf. A useful framework for evaluating several dimensions of digital library usage.
- Roxanne Missingham, What Makes Libraries Relevant in the 21st Century? Measuring Digital Collections from Three Perspectives (2003) http://www.nla.gov.au/nla/staffpaper/2003/missingham2.html.
- Thomas C. Reeves, Xornam Apedoe, and Young Woo, Evaluating Digital Libraries: A User-Friendly Guide (2003) http://eduimpact.comm.nsdl.org/evalworkshop/UserGuideOct20.doc.
- Tefko Saracevic, How Were Digital Libraries Evaluated? (2004) http://www.scils.rutgers.edu/~tefko/DL_evaluation_LIDA.pdf. An overview of previous digital library evaluation efforts.
Currently the COUNTER standards are dominant for measuring the use of digital collections, but they focus more on vendor-provided data than on collections produced by institutions.
- COUNTER (Counting Online Usage of NeTworked Electronic Resources) website http://www.projectcounter.org/code_practice.html. The COUNTER Codes of Practice are standards for measuring use of electronic journals, databases, and e-books.
- NISO, Standardized Usage Statistics Harvesting Initiative (SUSHI) website http://www.niso.org/committees/SUSHI/SUSHI_comm.html. Defines Web services- based harvesting of COUNTER usage data from different vendor platforms.
Usage data are somewhat limited when considered alone. When combined with input measures, output measures, or instructional data, they can help shed light on the effectiveness of a digital collection or digital library. Google Analytics (http://www.google.com/analytics/) provides tools to track where users come from and how they use a website.
Resources on collection evaluation methods, standards, and tools:
- Denise Troll Covey, Usage and Usability Assessment: Library Practices and Concerns (2002) http://www.clir.org/pubs/abstract/pub105abst.html. A good overview of research methods for studying collection use.
- International Coalition of Library Consortia (ICOLC), Revised Guidelines for Statistical Measures of Usage of Web-Based Information Resources (2006) http://www.library.yale.edu/consortia/webstats06.htm. Endorses COUNTER and SUSHI and provides guides for recording usage statistics.
- Managing Electronic Collections: Strategies from Content to User website http://www.niso.org/news/events/niso/past/Collections-06-wkshp/. Presentations from a NISO workshop held September 28-30, 2006 at Denver, Colorado. Presentations from day one “Understanding users and usage” and day two “Usage statistics wrap-up; practical collection and repository management” are especially useful.
- Association of Research Libraries, MINES for Libraries: Measuring the Impact of Networked Electronic Services website http://www.arl.org/stats/initiatives/mines. A web-based survey on user demographics and their reasons for using networked electronic resources.
- ANSI/NISO Z39.7-2004, Information Services and Use: Metrics & statistics for libraries and information providers--Data Dictionary (2004) http://www.niso.org/dictionary/. A data dictionary of terms pertaining to use metrics and statistics, includes measures for electronic resources; the main focus is on usage of resources in libraries.
Some collection assessment studies:
- Johan Bollen and Rick Luce, “Evaluation of Digital Library Impact and User Communities by Analysis of Usage Patterns,” D-Lib Magazine, v. 8, no. 6 (2002) http://www.dlib.org/dlib/june02/bollen/06bollen.html. Using server logs to assess the impact of a digital collection and understand the user community of a digital library.
- Christine Borgman, Evaluating a Digital Library for Undergraduate Education: A Case Study of the Alexandria Digital Earth Prototype (2002) http://www.sztaki.hu/conferences/deval/presentations/borgman.ppt. How users make use of digital collections and the effect of digital collections on students and instructors.
- Casey Jones et al, “Developing a Web Analytics Strategy for the National Science Digital Library,” D-Lib Magazine, v. 10, no. 10 (2004) http://www.dlib.org/dlib/october04/coleman/10coleman.html. A summary of NSDL evaluation efforts.
- Susan Musante, Evaluating MicrobeLibrary on Many Levels: Library Use, User Needs, Accessibility Issues, and Educational Impact (2004) http://nsdl.comm.nsdl.org/meeting/archives/2003/wiki/uploads/36/MicrobeLibrary_NSDL_2003_Presentation.ppt.
- Chris Neuhaus, Digital Library Evaluation: Measuring Impact, Quantifying Quality, or Tilting at Windmills? (2003) http://nsdl.comm.nsdl.org/meeting/archives/2003/wiki/uploads/36/nsdlevaluation_ 101503_eerl_2.1.ppt.
- Michael Organ, “Download Statistics: What Do They Tell Us? The Example of Research Online, the Open Access Institutional Repository at the University of Wollongong, Australia,” D-Lib Magazine, v. 12, no. 11 (2006) http://www.dlib.org/dlib/november06/organ/11organ.html.
- Kristen Fisher Ratan, Applications of Usage Statistics (2006) http://www.niso.org/news/events/niso/past/Collections-06-wkshp/MEC06-05-Ratan.pdf. How usage statistics are used for collection management purposes.
Last updated: 09/03/2008