D-Lib Magazine
May 2005

Volume 11 Number 5

ISSN 1082-9873

Influencing User Behavior through Digital Library Design

An Example from the Geosciences

Cathy A. Manduca, Ellen R. Iverson, Sean Fox
Science Education Resource Center
Carleton College
{cmanduca, eiverson, sfox}@Carleton.edu

Flora McMartin
MERLOT
<mcmartin@merlot.org>

Each of the elements of a digital library – its collections, the portals to those collections, and its supporting services – plays an important role in determining how the library is used and what impact it will have on users. In this article, we explore how the portal of a digital library can be designed to influence the behavior of its users. Our example is the Starting Point Digital Library (DL), an education digital library designed primarily for faculty who are teaching introductory level geoscience. Like many education digital libraries, we aspire to enhance teaching and learning in a realm of science [1, 2]. The specific goal of the Starting Point library is to increase faculty knowledge of the full range of teaching methods that can be used in their classes. With this knowledge they will be better positioned to design and implement introductory geoscience courses that support student learning. We draw on three powerful tools for designing and refining the library portal and its underlying information architecture: 1) techniques of user-centered design that engage users in design and testing [3, 4, 5, 6], research on the work needs and habits of targeted user groups [7], and evaluation techniques that use webmetrics to understand use patterns [8].

The Starting Point DL combines traditional digital library discovery through search and browse, with thematic portals that contain substantial content designed to motivate and facilitate use of the library collections. To successfully design the library requires an understanding of users' current work patterns and attitudes toward the library content. In our particular case, where we desire to bring information about teaching entry level geoscience contained in our library into use by geoscience faculty, it was critical to know how faculty learn about teaching, how they make decisions about what they do in class, and how they use the web in preparing for class.

To answer these questions, we undertook a research study of geoscience faculty and graduate teaching assistants who teach. The results of this study were used to design the Starting Point DL portal and its underlying information architecture. Walkthroughs and focus groups were then used to test and refine this design. Web metrics and evaluation studies we have done indicate that the design is supporting the anticipated user behavior and is leading to widespread discovery of pedagogic resources in the Starting Point DL.

Conducting Research as a Basis for Design

To obtain an in-depth look at how geoscience faculty think about and prepare for teaching, we conducted one-on-one interviews with a small sample of geoscience faculty. We were particularly interested in the kinds of resources they used to learn about teaching, the types of materials they needed to prepare for lecture and lab, and their use of the web in finding both information and materials. In addition, we explored the processes faculty use to design student activities and the role pre-existing activities designed by others plays in activity development.

Eight faculty members and one graduate student who were either currently teaching or had recently taught an introductory course were interviewed during the 2003/2004 academic year. The participants were drawn from a full range of institution types in Minnesota [9] and included both males and females, as well as faculty with different amounts of teaching experience. Interview data were collected through audio-recordings and from the researchers' written field notes, and were coded using the constant comparative method of qualitative analysis [10]. The results are validated by several studies of faculty's views on teaching [8, 11, 12].

A cognitive walkthrough [4, 13] of a prototype site was included in the interviews. This protocol was then used with 21 additional faculty to refine the Starting Point site design and to verify observations from the initial interview study. To determine if our observations of faculty use of the web were typical of a larger sample, we collaborated with the On the Cutting Edge project to include several questions in their survey of geoscience faculty in the United States [14]. This survey was administered by Statistical Research Center of the American Institute of Physics (AIP) to 5,700 faculty members and received 2,207 responses, thereby providing a robust sample of faculty behavior in the United States.

Results

Interview results showed that while faculty think about teaching in a wide variety of ways, have divergent philosophies of teaching, and employ very different approaches in their teaching, there are a number of commonalities in the ways that they learn about teaching and search for resources. Three ideas emerged that may be important in guiding the design of on-line resources and digital libraries for faculty: pedagogy and content, the role of the colleague, and preparation for teaching.

Pedagogy and Content: Faculty frequently think about teaching methods in the context of the material they teach. While many faculty have a general knowledge of teaching methods, they are most interested in the application of these methods to the specific topics they teach, and they prefer to learn about teaching methods within such a context. Examples of the use of methods to teach geoscience are particularly important as a bridge between their own work and a more abstract discussion of teaching methods. In site walkthroughs all thirty participants went immediately to such examples provided by the Starting Point DL. This result suggested that a design linking information about teaching methods to examples on teaching specific content will be more effective in both supporting discovery of the information on teaching methods and in placing it in a context that supports use.

The Role of the Colleague: Faculty colleagues play a critical role as sources for information about teaching and in validating the decisions faculty make about methodology and content. Seven of eight faculty we interviewed (87%) identified colleagues as a favorite source of information on teaching. When asked where they would start to develop a lab in an unfamiliar field, six of the eight interviewed (75%) indicated they would first ask a colleague. These results are consistent with the results from a larger, nationwide survey of geoscience faculty [14] and indicate that conversations with other geoscience faculty are mentioned more frequently as a source for information on teaching methods than are workshops, professional society meetings, publications, or on-line resources. The important role of faculty colleagues as a source for information on teaching suggests that faculty referral may be critical for building use of digital libraries like Starting Point, and that making visible use of faculty as authors of library content and portals may increase both interest and confidence in the site.

Preparation for Teaching: Faculty work within two time scales as they prepare their courses: 1) designing the entire course and 2) preparing for a single class. Faculty we interviewed talked about designing a course in the year, month, or weeks before the course takes place. This activity typically culminated in completion of the course syllabus. It was during this phase that faculty felt they might be able to make a large change in their preparation methods that would result in a change in the structure of the course or its assignments. Only two of the faculty (25%) we interviewed used the web to find information that assisted them in designing the structure of a class. Three of eight faculty (37%) reported looking for course syllabi either for inspiration or for validation that their course covered standard topics or was at an appropriate level. Only one faculty had looked for on-line pedagogic information. The nationwide survey of geoscience faculty confirmed that a relatively small fraction (10%) of faculty use the web to find syllabi, with an even smaller percentage of these using the web to find information about teaching (10%).

In contrast, faculty described preparing in the preceding minutes, hours, or days for an upcoming class. During this time they frequently seek materials to enhance an existing lecture or, less frequently, seek materials with which to design a new activity. All faculty interviewed and 87% of faculty surveyed use the web to download materials for lecture, most commonly images but also data and information. Table 1 shows materials identified by surveyed faculty as their favorite on-line resources. (1,592 responded to this question of the 2,207 who responded to the AIP administrated survey.) The faculty we interviewed located these materials by searching on specific geoscience topics or terms confirming the subject-based seeking behavior noted across the sciences [15]. Similar to other studies [16], a significant fraction of the faculty we interviewed (66%) and of those who responded to the survey (50%) are interested in looking for ideas for their teaching on the web.

Table 1. Favorite web-based teaching resources used by Geoscience faculty.

These data suggest that, to have a major impact on teaching methods, Starting Point would need to influence faculty during the design phase of their courses. This required a design for Starting Point that would capitalize on faculty use of the web to find materials for class as a mechanism for bringing them into contact with materials that could be used later to support their redesign of a course. The data also suggest that the major impact on teaching may come a year or more after a faculty member's first visit to the Starting Point library.

Designing a Portal to Influence Teaching

Based on the results of this research and on-going evaluation studies, the Starting Point DL project created several design features aimed at maximizing the library's potential to have an impact on faculty teaching methods. In particular the Starting Point site seeks to:

• Associate information on pedagogic methods with examples that are searchable by geoscience topic.
• Facilitate communication among faculty and colleagues regarding teaching methods and experiences.
• Link information that will strengthen the design of future courses with materials faculty find in the course of preparing for tomorrow's class.
• Capitalize on Google searches on scientific topics and terms to bring users to the site.
• Facilitate learning about teaching as faculty explore the Starting Point library.

A Digital Library Design Integrating Pedagogy and Content

The need to strongly tie pedagogic material to topical examples led us away from a traditional digital library interface where a single search engine overlays a broad collection. Even if such a library were to include both pedagogic and topical resources, it would not directly facilitate faculty learning about pedagogy in the context of specific topical resources. Instead we constructed a portal where the collection of topical examples is intermixed within a site structure explicitly organized by pedagogic topic.

The site is divided into pedagogically focused modules on particular teaching topics (e.g., Socratic Questioning, Teaching with Models, Field Labs). Each module provides an introduction and fully referenced guidance from colleagues on the particular teaching topic and then offers an embedded, searchable set of geoscience examples (e.g., specific modeling activities, ideas for particular field labs). Our example collection, pages with pedagogic guidance, and all pedagogic references are managed with standard digital library tools [17]. We anticipated that most faculty would enter the site from a web search on a geoscience topic that would likely lead them to one of these examples.

Because the examples are embedded within the pedagogic module, they act as a bridge between the geoscience subject matter and the pedagogy. The navigational elements on each page provide clear links to the surrounding pedagogic context, and a link to related pedagogy is included in the text of each example. We also anticipated that users referred by colleagues to the site would enter through the Starting Point home page or the top of a module describing particular pedagogy. For these users, we showcased the example collection and allowed users to immediately search the collection. Both search and browse functions facilitate the ability of faculty to find examples of a variety of methods addressing a single geoscience topic.

Throughout the site, pedagogic resources in our collection are presented in three ways: 1) as links within specific teaching examples, 2) as links within sections of the site describing teaching methods, and 3) as links in reference pages associated with sections of the site describing teaching methods. In all cases the resources are associated closely with the teaching examples, supporting the ability of faculty to find and use the pedagogic literature in a context in which they will understand their relevance and application.

The Colleague as a Trusted Source

Maximizing Use of Pedagogic Information

The primary information seeking behaviors of geoscience faculty identified in this study – web-based searching on a geoscience topic and colleague referral – were used to maximize the number of faculty who find Starting Point. All underlying digital library records are explicitly exposed for crawling by search engines. The titles, text and images in example pages intentionally include terms specific to the geoscience content designed to be aligned with likely faculty search terms. By including examples that incorporate images and span the range of geoscience topics taught at the entry level, we increase the opportunity for faculty to find the site when looking for materials for lecture – their most frequent use of the web.

Once faculty members are on the site, our goal is to convince them that this is a site worth exploring now or in the future. The site is designed to showcase the types of information available. For instance, example pages are designed to enable the user to learn that in addition to the materials they are seeking, there is information on goals, tips, and assessments that may assist them in using the materials. Similarly, the range of pedagogic topics covered by the site is displayed in the navigation on each page, and the example collection is searchable from each page as well. In this way, we hope to create a bridge between their most common immediate goal, preparing for class, and their use of the site to design a future course. By making it clear that pedagogic information is available and associating it with well-reviewed resources on specific science topics, we hope to build faculty trust in the site as a source of accurate and useful information on pedagogy.

Evaluating Site Use and Impact

To evaluate the effectiveness of our design of Starting Point, we asked three questions:

• Are users behaving as we assumed?
• Are users finding information on pedagogic methods?
• Is interaction with the site and its resources impacting users' teaching?

Standard server logs were collected and analyzed using Sawmill [18]. In addition to data directly observable in the logs, this tool follows standard practice in reconstructing users' sessions, allowing exploration of how users move across a site. Session reconstruction based on server logs has well documented limitations, as it extrapolates from queries to the server and not actual user behavior, which is often masked by caching and proxies [19]. In addition, we use data collected as part of the National Science Digital Library (NSDL) webmetrics pilot [20], which provided a web analysis service [21] that gathered statistics through the placement of a javascript 'web bug' on each page. This technique side-stepped many of the caching issues inherent in server logs. Both types of analyses revealed similar broad patterns of use across the site, thereby giving us confidence that known limitations in each technique were not significant factors in our analysis. Though we speak specifically to site traffic in November and December of 2004, the patterns described in this article are consistent with patterns found in earlier log analyses.

Information about geoscience faculty use of the site and its impact on teaching is based on follow up phone calls with interview participants and a web-survey that received 20 responses.

Expected and Unexpected User Behavior

Analysis of the site web logs confirms our hypotheses about user behavior. Eighty percent of Starting Point DL visitors arrive at the site via a search engine: 65% from Google, with 10% specifically from Google Image Search. While the key pages and teaching examples within our site are cataloged and discoverable through the Digital Library for Earth System Education (DLESE) and NSDL, only 1% of our visitors come from these two sites.

Our expectation that faculty would search on specific geoscience topics is supported by our web statistics. Thirty percent of the top 200 search terms used to find our site were clearly geoscience-related (e.g., geology, earth, igneous). Similarly, 24 of the 50 most popular search phrases were geoscience-related (e.g., soil horizons). As expected, these searches directed users deep into our site – only 25% of visitors started their visits on the site home page or the topic page of a pedagogic module.

Our information architecture was designed to entice faculty to move from these geoscience-specific teaching examples upward into areas of the site that explicate the pedagogy involved in the examples. We see several use patterns that indicate this design was successful. If we look in detail at common paths through the site, we find two distinct types of paths played out repeatedly. Typical examples of these paths are shown in Figures 1 and 2.

Figure 1. The typical paths of visitors moving from topical example into pedagogic content.
(Data from October and November 2004.)

Figure 1 illustrates key features of the paths users follow when moving from an example to the pedagogic information and to the site as a whole. Visitors enter the site through an example page, likely from a Google search for a term such as 'global warming project'. Those who stay on the site are likely to head to a related global warming page (linked within the text of the first page). From there visitors take various paths, but half leave this section of the site to explore other pedagogic topics. This leads us to conclude that users who make the initial decision that our site may be of interest also successfully find their way to a variety of pedagogic topics, even when their initial search brought them to a deeply embedded topical example.

We also anticipated that a significant number of users would be referred to the site by colleagues and would enter through the top of the site or through a particular pedagogic module.

Figure 2 highlights a path typical of users who found the site at the top level and proceeded to explore it more deeply. Visitors entered the site via the front page of the Socratic Questioning module. The most popular next page on this path is a 'How to Use' page linked prominently in the text of the front page. Although users took a variety of paths, the single most popular path, as shown in Figure 2, led them to examples of this pedagogic technique. Additionally 25% of those who entered on this general topic explored the entire site fairly extensively, visiting at least five if not more pages.

Figure 2. Typical path showing visitors starting from a top-level pedagogic page.
(Data from November 2004.)

Finally, our analysis reveals a large, unexpected, pool of users who reach the site by searching on a pedagogic topic. Ten of the top twenty-five search phrases relate to pedagogic concerns. For example, the most popular search phrase bringing people to the site is 'Socratic questioning', which leads to the path described in Figure 2. It is unclear who this audience is, as our interviews indicate that searching directly for pedagogic information is not a behavior typical of geoscience educators. This behavior typifies several thousand visitors per month, which suggests this is a large audience and not a small subset of geoscience faculty. These users may be educators from a wider spectrum of disciplines who are interested in pedagogy.

Impact on Teaching

An in-depth examination of the impact of the site on teaching is still in the future. However, site usage serves as a type of a surrogate measure, assuming that, for example, those users who spend more time at the site may be more likely to use an idea or adopt an activity to try in their class.

The site's usage statistics suggest that users are finding the site of value: site traffic has increased steadily since the site's inception, growing to roughly 9,000 visitors per month in 2004. Fifteen percent of visitors spend ten minutes or more on the site and approximately ten percent return several times per month.

Comments from our web-based survey and from walkthrough participants suggest continued impact on teaching. Fifteen of twenty respondents to the survey identified specific activities or teaching methods that they planned on using in a current or upcoming class. Seventeen of twenty-one participants in walkthroughs that did not involve structured interviews singled out, by name, activities that they either bookmarked, emailed to themselves, or made a note to use in the near future. In follow up phone calls, six out of eight interview participants described specific scenarios where they either had used content from the site or had incorporated it into an upcoming class.

Informal discussions with geoscience faculty at various professional society meetings suggest that faculty are only beginning to be aware of the site. It seems likely that site impact will increase as awareness of the site grows and faculty have the time needed to incorporate ideas into their courses.

Conclusion

The integration of research studies, user-centered design techniques, and evaluation studies provided powerful guidance in the development of the Starting Point DL portal as a tool for professional development of geoscience faculty. The research study we completed showed ways in which the site design could build on the existing work habits of our users to bring them in contact with new information that they did not traditionally look for on the web. Further, our research suggested ways in which we could present material that would increase faculty confidence and thus their inclination to use the Starting Point site. Walkthroughs refined the design of the site and also our understanding of user behavior with the actual library portal. Evaluation studies are helping us to generalize this understanding to the full spectrum of our users while linking the use of the site to its impact on teaching – the overarching goal of the project and its funders.

Acknowledgment

This work is funded by the National Science Foundation under the grant DUE 0226243.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

[1] Manduca, C. A., F. McMartin and Mogk, D. Pathways to progress: Vision and Plans for Developing the NSDL: Report to the National Science Foundation. 2001. Available at <http://nsdl.comm.nsdl.org/meeting/archives/smete/meetings/grantees0901/whitepaper.pdf>.

[2] Zia, L., The NSF National Science, Mathematics, Engineering and Technology Education Digital Library (NSDL) Program: A Progress Report. D-Lib Magazine, October 2000. Available at <doi:10.1045/october2000-zia>.

[3] Armitage, L. and Enser, P.G.B. Analysis of user need in image archives. Journal of Information Science, 23(4), 1997, 287-299.

[4] Carroll, J. (ed) Scenario-based design: Envisioning work and technology in systems development. John Wiley & Sons, New York, 1995.

[5] Hackos, J.T., User and Task Analysis for Interface Design. John Wiley & Sons, Inc., 1998.

[6] Nielsen, J. Designing Web Usability: the Practice of Simplicity. New Riders Publishing, Indianapolis, 2000.

[7] Borgman, C. et al. How geography professors select materials for classroom lectures: implications for the design of digital libraries. Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries. ACM Press, New York, NY, 2004, pp. 179-185.

[8] Punziak, J., McMartin, F., and Agogino, A., Building a Digital Learning Community for Faculty on the Internet. (in Proceedings of ASEE 2000, Paper #3630, ASEE, 2000). 2000.

[9] McCormick, A., Ed., The Carnegie Classification of Institutions of Higher Education, 2000 Edition. The Carnegie Foundation for the Advancement of Teaching, Stanford, CA, 2001.

[10] Glaser, B and Strauss A. Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, Chicago, IL,1967.

[11] Foertsch, J.A., Millar, S.B., Squire, L.L. and Gunter, R.L. Persuading Professors: A Study of the Dissemination of Educational Reform in Research Institutions. Univeristy of Wisconsin, LEAD Center, Madison, 1997.

[12] Rogers, E.M. Diffusion of Innovations. 5th edition, Free Press, 2003.

[13] Nielsen, J. Heuristic evaluation. In Nielsen, J., and Mack, R.L. Eds., Usability Inspection Methods. John Wiley & Sons, New York, NY, 1994.

[14] Macdonald, R.H., Manduca, C.M., Mogk, D.W. and Tewksbury, B.J., Teaching Methods in Undergraduate Geoscience Courses: Results of the 2004 On the Cutting Edge Survey of U.S. Faculty. Journal of Geoscience Education, in review.

[15] Bruner, J. The Process of Education. Harvard University Press, Cambridge, Mass, 1960.

[16] Chang, A., Matyas, M., Gough, N., George, Y. I Came, I Found It, I Used It, and It Made a Difference. American Society for Microbiology, 2004. <http://biosciednet.org/project_site/BEN_Survey_Article_October_2004.pdf>

[17] Fox, S., Manduca, C.M., Iverson, E. Building Education Portals atop Digital Libraries. D-Lib Magazine, January, 2005. Available at <doi:10.1045/january2005-fox>.

[18] Sawmill website. Available at <http://www.sawmill.net>.

[19] Spiliopoulou. M., Bamshad, M., Berendt, B. and Nakagawa, M., A Framework for the Evaluation of Session Reconstruction Heuristics in Web-Usage Analysis. INFORMS Journal on Computing, Vol. 15, No. 2 (Spring 2003), 171-190. Abstract available at <http://www.extenza-eps.com/extenza/loadHTML?objectIDValue=14445&type=abstract>; access to full text is available for subscribers only.

[20] Jones et al., Developing a Web Analytics Strategy for the National Science Digital Library. D-Lib Magazine, October 2004. Available at <doi:10.1045/october2004-coleman>.

[21] Omniture Site Catalyst. Available at <http://www.omniture.com>.

(On February 2, 2006, an acknowledgment and a disclaimer were added to this article at the request of its authors.)

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | Next article
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/may2005-fox