D-Lib Magazine
July/August 2005

Volume 11 Number 7/8

ISSN 1082-9873

JCDL Workshop Report

Studying Digital Library Users in the Wild

Michael Khoo
DLESE Program Center
University Corp. for Atmospheric Research
Boulder, Colorado, USA
<mjkhoo@ucar.edu>

David Ribes
Sociology and Science Studies
University of California-San Diego
San Diego, California, USA
<dribes@ucsd.edu>

Introduction

As digital libraries (DLs) continue to mature into complex networked information and knowledge tools, and become accessible to ever more diverse groups of users, library implementation and use are emerging as important research problems. How might we understand the ways in which the current generation of DLs have been designed and are used in real-life contexts today, and how might we apply our understandings to future generations of DL design?

Workshop Theme and Structure

The JCDL workshop "Studying Digital Libraries in the Wild" was convened with the intent of supporting constructive dialogue amongst DL researchers involved in the qualitative analysis of DL use. The workshop call for participation resulted in the submission of a number of high-quality position papers that addressed a range of theoretical and methodological issues, and the papers were all highly relevant to the qualitative, naturalistic, and longitudinal study of DLs 'in the wild.' A total of eleven position papers were accepted, and these are available in this issue of D-Lib Magazine along with this workshop report.

Workshop participants were asked to treat DLs as sociotechnical artifacts (Bishop et al., 2003). These DLs represent complex mixtures of people, practices and technologies whose design, implementation and use bring together a wide range of sponsors, developers, and users, in a range of institutional and social contexts (Bijker, 1995). The development of sociotechnical artifacts can be unpredictably influenced by social and technological contingencies (Bijker et al., 1987). If DL development is affected in the same way, then the identification and analysis of the dynamics of these contingencies and of how they affect the behaviors of DL users 'in the wild' (Hutchins, 1995) becomes a crucial component of overall DL design.

The workshop consisted of a series of panels and short presentations that focused on research methodologies addressing the issue of sociotechnical complexity. In talks reflecting their position papers, participants described 'works in progress' and outlined case studies – the latter in particular providing excellent fodder for stimulating discussions regarding theoretical and methodological approaches, practical issues such as relationships with research subjects and sponsors, and general 'tales of the field' (van Maanen, 1988). Common themes and questions about qualitative user evaluation emerged over the period of the workshop, including:

• Flexibility of approach
• Archiving and sharing data
• Communication and intervention

Flexibility of Approach

Perhaps the most obvious theme of the workshops was the diverse range of research methods that participants had applied in their own work. The main methods introduced are summarized here. (The brief sketches in this report are, of course, incomplete descriptions of each participant's research; see the archived position papers for more detailed accounts).

Adams, discussing tool implementation, described the use of grounded theory in studies of user authentication and privacy, and the introduction of DL technologies into clinical settings. She described how grounded theory had identified a number of unintended social consequences following the introduction of these technologies, especially in the social and institutional orders of the clinical settings.

Notess, who also focused on the process of inductive model generation, provided a detailed description of the application of Contextual Design methods to the study of music DLs, and emphasized the need for such studies to become both more widespread and also more systematic within the DL community (for instance in the form of data and model sharing).

The papers by Buchanan et al., and Hinze (both presented by Buchanan), discussed methodological and practical issues (as well as the unexpected findings) that can arise when human-computer interaction (HCI) research analyzes the fine grain of everyday user interactions. These researchers have analyzed news alert services and the search needs of humanities scholars, and in the process, they have questioned some common HCI assumptions regarding the ability of HCI methods to uncover what the user 'really' knows about, or wants from, a particular technology. Special care is needed here both in framing the research question(s) and in applying these to subjects of the evaluation.

Wolf, describing the early stages of research with several educational DLs, noted that a particular problem with user testing and evaluation is something we, as researchers, sometimes forget, and that is that many research subjects may not even have considered DLs as technologies.

Speaking directly to ethnographic experience, the richness and complexity of real world practices was illustrated by Cunningham, who described how ethnographic observations conducted outside the laboratory (for instance, in music stores and people's homes) revealed how people used the containers of digital media (CD cases) to personalize their social spaces and also to communicate with one another.

Weedman provided a wide-ranging review of a number of approaches that she has brought to bear on her field data, including the sociology of science, the sociology of art, STS, practice theory, and design theory, and she described how these have been applied to a series of different field sites, including image digitization projects and metadata design.

Ribes, drawing on Actor Network Theory, described the complex sociotechnical ecology of ethnographic study itself, recounting several reflexive moments from his own ethnographic observations in a number of field sites when he realized that his presence within his field sites had triggered of a series of unintended social consequences within those sites. The conclusion to be drawn here is that researchers cannot avoid having an influence on their study sites.

A number of the workshop participants described the use of computational tools to archive and analyze data. Coding tools such as NVivo (http://www.qsrinternational.com) and ATLAS.ti (http://www.atlasti.de) were popular choices for the inductive coding and analysis of texts (such as transcripts of conversations, e-mail, threaded discussion, documents, etc.). Several researchers also described how they also used tools to parse (as well as to code) texts. Shumar argued for considering online technology users as being bound together in symbolic communities, constructed and mediated through discourse, an analysis that opens up DL research to a range of anthropological theory; he studies the boundaries of such communities with Netscan, a tool that analyzes discussion threads and spaces. Khoo described the use of computational text analysis tools to identify the presence of regular patterns in spoken and written discourse, patterns that could point to the existence of differences in tacit understanding amongst different DL groups such as developers and users. Notess (see above) shared how he has used contextual design to generate models and diagrams of user behaviors that can then be used to inform design and also can be represented back to users.

In discussion, the participants moved freely between their own theoretical and methodological 'home territory' and that of other participants, and they were quick to point out that each method had advantages and disadvantages; no one method is perfect for all locales and research questions. As discussions progressed about juxtaposition of methods some common issues surfaced, including questions of scale, resolution and granularity, and the need to balance (where appropriate) macro and micro, and inductive and deductive, approaches. Here, using pragmatic 'tool box' approaches that combined elements of different methods was often seen as an appropriate response to the granularity question – indeed, methods such as contextual design can be seen as attempts to formalize the tool box approach – although also noted was the danger that the use of individualized suites of methodologies could reduce the ability to generalize research findings and share data.

Another significant issue that emerged during the discussion was that of the importance of the notion of 'practice' to many theoretical and methodological approaches (for instance, in the form of practice theory, communities of practice, etc.). An interest in practice and the hands-on activities and know-how of actors in particular circumstances often transcended (or perhaps united) disparate theoretical and methodological frameworks, and permitted an understanding of behaviors reflecting 'pre-digitalization' attitudes to new technologies, local adaptations of technologies, work-arounds, rejections of technologies, and so on. The notion of a 'community of practice' was mentioned in several presentations and discussions, pointing to the consideration of collectives of actors as defined by the things they do rather than by their 'official' organizational positions.

Archiving and Sharing Data

As was mentioned in the previous section, qualitative data analysis is increasingly supported by software tools such as NVivo and Atlas.ti, which allow for the inductive creation of codes by which data may be marked-up. These methods augment traditional qualitative data methods with digital technologies, and it was noted that this digital functionality is reproducing, in the context of DL qualitative research, the more general concern for data sharing and interoperability in scientific circles raised by the Atkins report (Atkins, 2003). That is, while DL researchers may be collecting, archiving, and analyzing qualitative data in ever-increasing quantities, little is being done to enable data-sharing across research communities.

This prompts the question of how we may archive and exchange our digital data, (for example, for historical record, for comparative and meta-analysis, or for future re-use). While software packages such as NVivo and Atlas.ti can support collaborative and distributed projects – allowing the merging of coding categories – as it stands, there are no wide-scale efforts for the preservation and exchange of DL qualitative data. Further, the nascent state of our endeavors means that many aspects of such work are presently under-thought (issues of data accessibility, data security, and data privacy, for instance, remain to be worked on). In this regard, the cyberinfrastructure initiative has recently turned its attention to the social sciences, and it is hoped that research aimed at developing such tools will be conducted in the future (see, for instance, http://vis.sdsc.edu/sbe/reports/SBE-CISE-FINAL.pdf).

Communication and Intervention

While qualitative DL research can provide rich descriptions of cultural practices and uses of technical artifacts, the complexity of these findings can make communicating them with diverse audiences such as DL developers, policy makers and end-users difficult. Additional framing is often necessary in order to present results. Two lines of discussion emerged regarding the question of communication: first, developing strategies for communicating with those not familiar with qualitative research, including the possibility of novel representational forms; and second, establishing forums to support communication amongst qualitative DL researchers, and between researchers and the outside world.

With regard to developing communication strategies, qualitative research needs to emulate quantitative research, which has traditionally used visual expression, such as graphs or charts. Visual representations have the advantage of quickly communicating large sums of data in synoptic fashion, and a number of visualizations have become standardized across fields, facilitating inter-disciplinary decoding (e.g., a bar chart). There is room to develop similar representations for sociotechnical research. Here, researchers should not be limited to traditional representational mechanisms, but should also consider such forms as concept maps, social networks, organizational charts, etc.

With regard to establishing communication forums, there was a general feeling that qualitative DL research is often peripheral to those within the wider DL research community. To strengthen the position of such research, there is a need for a digital space for the archiving of qualitative data and for the discussion of methodological and theoretical issues and case studies. Such a space would support the development of 'toolbox' approaches to DL research, permitting comparison of research sites and findings, and supporting the emergence of more general theory and understanding, as well as presenting rich theoretical and methodological accounts, backed up by 'thick' case studies. Such a forum need not necessarily be limited to DL research, but could include examples of research into other information, communication, and knowledge technologies. It could also pay attention and make contributions to sociotechnical and STS studies outside of DL research.

Following on from the workshop, several initiatives will be undertaken to establish such a space and support the qualitative DL research community. The first step is the archiving of this report and the workshop position papers here in D-Lib Magazine. We will also mirror these documents on a Plone server at the Digital Library for Earth System Education (DLESE: http://www.dlese.org/cms/qdl/). This mirror is expandable and editable, and researchers will be able to upload and archive papers, workshop reports, etc. in the future. The Plone site at DLESE will function as a repository for qualitative DL research 'gray literature,' and it is hoped that this function will prove particularly useful for researchers who do not have access to the resources necessary to generate formal peer-reviewed publications for DL conferences and publications. The Plone site at DLESE will also have wiki-like capabilities to support the posting of announcements, etc. Establishing a qualitative DL research list is also planned, again on a server at DLESE, for announcements and the archiving of relevant documents.

Finally, a number of participants expressed enthusiasm for holding another workshop on user evaluation. The upcoming European Conference on Digital Libraries (ECDL 2006) is a possible venue, which would afford European colleagues greater opportunities to attend and present their research. A 'traveling' qualitative DL workshop is also a possibility.

Conclusion

The workshop position papers and discussions spanned a wide range of evaluation approaches. Rather than pointing up any weakness in the sociotechnical approach, this variety of approaches underlined the complexity of the research question(s) being addressed. The workshop also illustrated the creative possibilities inherent in dialogs between those using these approaches, as was evidenced in the lively exchanges that occurred during the various presentations.

Judging by the enthusiasm of the attendees, the outlook for qualitative DL research is favorable. Hopefully, as DL research in general matures and shifts from technological development to deployment and use, DL stakeholders will become progressively more amenable to complex and situated analyses of use. DLs are community efforts, involving an enormous diversity of domains, as well as traditions of knowledge acquisition and management, and in the face of this diversity, drawing on a full range of qualitative research approaches seems necessary and productive.

Workshop Position Papers

Adams, Anne. "Grounded Theory: Case Studies and Methodological Issues," (http://www.dlib.org/dlib/july05/khoo/01_adams.pdf).

Buchanan, George, Ann Blandford, Jonathan Rimmer, Claire Warwick. "Usability Challenges in Digital Libraries for the Humanities," (http://www.dlib.org/dlib/july05/khoo/02_buchanan.pdf).

Cunningham, Sally Jo. "People and their Music," (http://www.dlib.org/dlib/july05/khoo/03_cunningham.pdf).

Gazan, Rich. "Digital Library Evaluation: A Longer View," (http://www.dlib.org/dlib/july05/khoo/04_gazan.pdf).

Hinze, Annika. "Issues in Usability Studies for Alerting in Digital Libraries," (http://www.dlib.org/dlib/july05/khoo/05_hinze.pdf).

Khoo, Michael. "Technological Frames and Language-games: Understanding Tacit Knowing," (http://www.dlib.org/dlib/july05/khoo/06_khoo.pdf).

Notess, Mark. "Using Contextual Design for Digital Library Field Studies," (http://www.dlib.org/dlib/july05/khoo/07_notess.pdf).

Ribes, David. "The Positions of the Social Scientist: Social and Technical Acts of Intervention," (http://www.dlib.org/dlib/july05/khoo/08_ribes.pdf).

Shumar, Wesley. "Ethnography at the Math Forum," (http://www.dlib.org/dlib/july05/khoo/09_shumar.pdf).

Weedman, Judith. Designers, "Clients, and Knowledge Production," (http://www.dlib.org/dlib/july05/khoo/10_weedman.pdf).

Wolf, Alan J., Ellen Iverson, Flora McMartin, Cathy Manduca, Glenda Morgan. "Understanding Faculty Needs of Digital Resources and Libraries," (http://www.dlib.org/dlib/july05/khoo/11_wolf.pdf).

References

Atkins, D. E. C. (2003). Revolutionizing Science and Engineering Through Cyberinfrastructure: Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure, National Science Foundation.

Bijker, W. (1995). Of bicycles, bakelites and bulbs: Toward a theory of sociotechnical change. Cambridge, MA: The MIT Press.

Bijker, W., T. Hughes, and T. Pinch, The social construction of technological systems. Cambridge, MA: The MIT Press, 1987.

Bishop, A., N. Van House, and B. Buttenfield, DL use: Social practice in design and evaluation. Cambridge, MA: The MIT Press, 2003.

Hutchins, E., Cognition in the wild. Cambridge, MA: The MIT Press, 1995.

van Maanen, J. (1988). Tales of the field: On writing ethnography. Chicago: Chicago University Press.

Top | Contents
Search | Author Index | Title Index | Back Issues
JCDL 2005 Conference Report | In Brief
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/july2005-khoo