D-Lib Magazine
February 2004

Volume 10 Number 2

ISSN 1082-9873

Supporting Member Collaboration in the Math Tools Digital Library

A Formative User Study

Nicole Shechtman
<nicole.shechtman@sri.com>

Mark Chung
<mark.chung@sri.com>

Jeremy Roschelle
<jeremy.roschelle@sri.com>

SRI International

Abstract

In this paper, we discuss a user study done at the formative stage of development of a Math Tools developers' community. The Math Tools digital library, which aims to collect software tools to support K-12 and university mathematics instruction, has two synergistic purposes. One is to support federated search and the other is to create a community of practice in which developers and users can work together. While much research has explored the technical problem of federated search, there has been little investigation into how to grow a creative, working community around a digital library. To this end, we surveyed and interviewed members of the Math Tools community in order to elicit concerns and priorities. These data led to rich descriptions of the teachers, developers, and researchers who comprise this community. Insights from these descriptions were then used to inform the creation of a set of metaphors and design principles that the Math Tools team could use in their continuing design work.

Introduction

The present National Science Foundation funded effort to create a National Science, Technology, Engineering and Mathematics Digital Library (NSDL) builds upon a long-standing vision of the purposes a national digital library dedicated to teaching and learning could serve (National Research Council, 1998). A primary purpose of the NSDL is to support federated search (NSDL, 2003), thus simplifying an individual instructor's effort to find relevant, high quality resources. A secondary purpose is to provide shared workspaces for the collaborative development of better teaching practices and resources [1].

The two purposes are synergistic: federated search is valuable only when very good quality materials can be found in the repository; and potential educational contributors need support to create high quality digital materials. Such support must come primarily from peers in order to be scaleable and sustainable. An important problem in the construction of an educational digital library, then, is the design of supports for collaboration among members to create high quality teaching resources.

A particular instance of this problem has arisen in the Math Tools project [2], a collection within the NSDL that aims to collect software tools (e.g., applets, graphing calculator programs, etc.) to support K-12 and university mathematics instruction. The Math Tools team decided that a priority for their effort would be to attract developers, teachers, and researchers to a "developer's area" of the Math Tools site, and to foster the formation of a community that would support each other's work-in-progress in the creation and adaptation of software tools and resources.

Whereas federated search is a fairly well-structured technical problem, growing a creative, working community around a digital library (DL) is not (Borgman, 1999). In a previous project, the authors of this article found that high quality math tools can be produced when developers, teachers, and researchers work together (Roschelle et al., 2000). Rather than take the risk of employing a strategy to "build something and see who comes," the Math Tools team asked us to conduct a user study. Although user studies are becoming increasingly common in digital library design, most aim at improving and evaluating usability of varying interfaces to core functionality already specified, architected, and implemented (see Project i-DLR, 2003). A different kind of user study was called for here—a formative study that would help the Math Tools team make fundamental decisions about what core functionality the developers' area should have. The aims of such a study have been described in an early digital libraries workshop report (Borgman, 1996, p.6) as:

1. Discovering which functionalities user communities regard as priorities.
2. Developing basic analytical categories that influence the design of system architecture.
3. Generating integrated design processes.

To discover desirable functionalities, we surveyed and interviewed Math Tools members who expressed interest in the forthcoming developers' area. The survey allowed us to summarize their concerns and priorities. To describe basic analytical categories, we used interview data to create fictitious "personas" representing important constituents in a memorable and generative way. In support of an integrated design process, we extended our understanding of the data to create a metaphor and formulate some principles that the Math Tools team could use in their continuing design work.

Method

We surveyed Math Tools registrants who had identified themselves as interested in receiving updates about the Math Tools developers' area. The survey was administered via the web in June 2003. Participation in the survey was voluntary and uncompensated. Of the roughly 155 registrants given the opportunity to respond, 56 completed the survey within two weeks of the request. We have reason to believe that these 56 were the most active, engaged, or interested registrants and therefore provided a realistic representation of those who would take part in a thriving Math Tools user community.

The survey inquired about gender, geographic location, occupation, primary occupational setting, and years of professional experience. The survey also asked whether the respondents developed software themselves and, if so, in what context. In addition, the survey asked what the developers hoped to get from being members of the Math Tools community.

Once the survey had been completed, we conducted follow-up, semi-structured interviews with 10 survey respondents who had indicated that they developed software. Demographically, these 10 interviewees were representative of the 20 developers surveyed overall. Interview questions pertained to professional and technical experience, the kind of software developed, development processes, the developer's software users, and thoughts about the Math Tools community.

The interviews were conducted over the telephone by three members of our design team: a social scientist, an education researcher who was formerly a teacher, and a software developer. This multidisciplinary approach to interviewing helped us to obtain a balanced and rich perspective.

Who is the Math Tools Community?

The survey revealed that professionals attracted to the Math Tools developers' community are primarily teachers and other individuals involved in various aspects of K-12 education. Table 1 summarizes the survey findings.

About half of the Math Tools community is composed of teachers in K-12 schools (and international equivalents). The rest of the user community is made up of software developers, graduate students in educational technology, education researchers, graphic designers, and a variety of educational administrators and support staff. The survey also showed that the Math Tools community was generally quite experienced, as the majority of professionals had more than 5 years of experience in their field.

About one-third of the community members developed their own software. Of the 56 members surveyed, 20 (36%) reported that they develop their own math software. The majority of these developers (75%) created software not as their primary profession or studies, but in support of their primary profession or studies. About one-third developed math software professionally, and many others developed math software as a hobby.

The developers we interviewed shared several characteristics. All had a passion for math, for making math understandable, and for math technology. Most, even the professional developers, were self-taught programmers. In general, those interviewed sensed that math software was not a venue for making a lot of money. Most of the developers also had in common that they were solo pioneers working in isolation.

The teachers we interviewed described themselves as nontraditional, creating classroom environments where they spend less time lecturing and more time acting as facilitators as students make mathematical discoveries themselves through hands-on work. In describing their teaching style, these teachers used phrases such as "enlightened and traditional," "casual," and "cooperative learning." The teachers interviewed also generally worked with top-tier, highly motivated and advantaged students.

Survey respondents were widely distributed throughout the US, and a surprising percentage of them were from locations outside the US (surprising because NSDL is national rather than international in orientation). The 41 US respondents represented 22 different states, no state of which had more than four respondents. Beyond the US, 27% of the respondents were from Canada, Asia, Europe, and Australia. One of our interviewees who spoke to us from England revealed some of the challenges he encounters as an international Math Tools user. For example, both math and education jargon differ across the countries, so he sometimes finds himself needing to figure out how to translate. (Even the American word "math" is "maths" in Britain.) There are also significant differences in curriculum structure, which can make searching a challenge.

Users Concerns and Priorities

In the survey and interviews, it became apparent that users' concerns and priorities were centered around searching for tools in and publishing software to the Math Tools DL, as well as participating in a community of practice.

Searching and Publishing

Across the board, everyone interviewed wanted a digital library where they could easily find math software. (Two developers likened searching the Internet for math tools to "finding a needle in a haystack.") Almost everyone reported wanting to get ideas from the Math Tools library. Teachers were looking for tools to use as-is, to augment, or to rewrite for their own classroom use. Professional developers were interested in understanding the space of tools available so that they could learn where their products would best fit. Both teachers and professional developers were concerned about "reinventing the wheel," putting too much effort into writing software that has already been done well. Researchers expressed interest in understanding the space of tools available in order to help inform their research.

In addition to finding tools, developers also wanted to be able to put tools they had developed into the library. The teachers and hobbyists noted how much satisfaction they would get from seeing others benefit from their hard work. The professional developers, too, were interested in finding ways to make their names and software visible to the user community at large.

Overcoming Isolation

In the interviews, both professional developers and teachers expressed feelings of "isolation." Integration of technology into the math classroom is still a relatively new and unexplored domain. Developers passionate about doing this often find themselves working solo in their pioneering ventures. Professional developers tended to be well connected with other developers, such that finding technical support was straightforward. However, they reported two other types of isolation. First of all, it was difficult to find other developers who were interested in collaboration specifically for math tools. Second, and much more critically, they felt isolated from classroom experience. Professional developers reported being out of touch with the needs and everyday realities of teachers and students as well as with the issues involved in software deployment in schools. They had little access to teachers who could give them ideas, feedback, and perspective on the classroom.

Likewise, teachers reported that they often found themselves working alone in the midst of disinterested administration and peers. Typically, the teacher's administration was somewhat financially supportive, but the administration did not truly understand what the teacher was doing. At the same time, the teachers' local colleagues tended to have little training or interest in technology, and thus were not interested in sharing or collaborating.

Thus, the theme of isolation has two components: a lack of local colleagues who can support or provide feedback on one's work and the difficulty of finding and working with potentially more supportive people from a different geographic locale.

Talking with Others about a Software Effort

The survey revealed that a majority of Math Tools users not only desire a digital library where they can find and publish math tools, but also hope to become part of a community in which they can communicate with others about math tools. While 84% of respondents wanted to come to Math Tools to get software they could use, 70% were interested in being part of a community of teachers who use math software. A significant proportion was also interested in being part of a community of developers and teachers willing to try out new software. Other respondents were interested in discussing software they were presently developing.

The interviewees described many social functions they wanted the Math Tools community to serve. Teachers, for example, wanted to know what other teachers were finding useful and how they were implementing software. As one teacher described it, "I'd like a meeting place to talk about what works." Professional developers wanted a place where they could talk to teachers to get the teachers' perspective on math and the classroom, as well as feedback on the software the developers offered. Some of the professional developers hoped to find teachers who could test the developer's software in classrooms and become participants in the iterative design process.

The issue of technical support also came up in the interviews. Since many of the teachers did not have access to adequate technical support, they wanted a place where they could ask technical questions about math software and programming, or even find help getting started as a programmer. The professional developers, working at a different level of proficiency and complexity, did not feel Math Tools was an appropriate place for them to discuss their own technical issues. They would, however, be able to offer technical assistance to others.

Researchers had yet another set of concerns. The researchers with whom we talked were deeply connected to both the world of teachers and the world of technology. They were in a position to see the gaps between these worlds and the opportunities for cross-pollination. They were aware of technology that teachers could but generally do not use, as well as the process of technology design that is often uninformed about actual classroom teaching practices. The researchers were interested in facilitating dialogs between these groups and opening up a new space of collaborative opportunities. Of course, bringing these different types of users together in one community presents certain challenges. Teachers and professional developers speak different languages and work in different professional worlds. While Math Tools could provide a venue for teachers and developers to meet, it may also need to facilitate professional connections and the process of interdisciplinary work.

Personas

The data collected in the interviews were used to create a set of user personas. A user persona is a design tool that helps define and characterize the users of a technology (Cooper & Reimann, 2003). It is a fictitious character sketch based closely on information gathered from real users. Personas help to contextualize and support design, grounding creative and speculative thinking in memorable character portraits. Personas can, for instance, bring use cases to life. Our data suggested five personas, each a composite category of a subset of the interviewees. The personas were:

1. Tom the teacher and experienced developer. Tom develops math software to help support teaching and learning in his own classroom.
2. Drew the professional developer. Drew develops math software tools as his profession.
3. Roger the researcher. Roger is concerned with educating math teachers and building community around technology. He develops technology that helps streamline math tool use for teachers.
4. Tammy the teacher and inexperienced developer. Tammy is an experienced math teacher who is technology savvy and wants to learn how to develop her own software.
5. Harry the hobbyist. Harry's hobby is developing math software in his spare time.

The first three of these personas are described in more detail below.

Tom the Teacher

Tom teaches geometry and calculus in a private school in West Virginia. Having taught for over 20 years, he is quite passionate about math and using technology to help teach math. He works with relatively bright and socio-economically advantaged students. Though his school administration doesn't truly understand what Tom is doing with the technology, they have bought his classroom 15 iMacs, and they provide the funds to send him to one conference a year. Tom's teaching style is focused around students working together to solve problems collaboratively. Math software plays an integral role in this. Tom finds that interactive software can help students learn to solve problems on their own, visualize and discover mathematical concepts and, most importantly, become engaged with mathematics.

Tom has three sources for math software. One source is his own stockpile that he's accumulated over the years. Another source is finding what he needs on the web. Finally, when he can't find something that has already been written, he writes his own. Tom has been teaching himself to program since he got an Apple IIe in the early 80's and has most recently been working on mastering Java. He showed us an applet on his homepage that illustrates Riemann's sums. He also showed us a Perl script he wrote to generate worksheets on derivatives.

Tom is excited about the Math Tools site. He would be grateful to have a digital library where he could easily find software relevant to his classes. He would also find it rewarding to have a venue for sharing his software so he could see other teachers use it and benefit from it. But he is interested in much more than the library itself. Feeling a bit isolated these days, he is intrigued about the prospect of becoming involved in a community of like-minded innovative teachers. He would highly value having a place to discuss what software is available, what software works in the classroom, and the occasional technical problem that comes up in his programming.

Drew the Developer

Drew talked to us from his home office in Orson, Michigan. His career has been somewhat circuitous. He graduated from college in 1982 with a major in math education, but instead of becoming a math teacher, over the years he took programming jobs at three different hardware companies. All along, however, he knew that his true passion was math and finding ways of making math understandable to others. Five years ago, he became an independent contractor, and is presently working under contract to a math software company. He has completed various projects, mostly developing math tools with one particular company, but he has developed some corporate training software as well. What he would like to do now is focus exclusively on developing math software.

As he begins to create a business plan for developing the math software, Drew has a few issues on his mind. First of all, he wants to get a handle on the range of tools already available in order to determine where best to concentrate his development efforts. Second, he is concerned about how he's going to get his software tools out to teachers and how to gain enough visibility in the math tools market that someone will be willing to pay him for the software he develops. Most importantly, he feels out of touch with what actually goes on in classrooms. He needs to explore how to develop tools that teachers will actually find relevant and useful in their classrooms, tools that will have an impact. Furthermore, he will need feedback from teachers as they try out his software.

Drew is glad to have found Math Tools. It has the potential to be a venue where he could get ideas, gain visibility for his software, interact with teachers, and obtain feedback about his software. Drew has much that he could offer the community, as well. He has invaluable software and technical expertise to share. He is not necessarily interested in being an active discussion leader or participant, but he would like to keep his eye on what's going on and contribute whenever he had the necessary knowledge and time.

Roger the Researcher

Roger talked to us from his office at a major state university in Washington State. He is a sixth-year Ph.D. student finishing up his dissertation in educational technology research. Before beginning his graduate work, he spent 10 years working first as a math teacher and then as a programmer at a small educational software company. In his experience in both schools and industry, he has observed that math teachers have large gaps in understanding the technology available to them, and developers do not often provide software most suitable to teachers' needs. Roger wants to find ways to bridge these gaps. He studies pedagogical issues related to integrating software into the math classroom and develops systems for helping teachers who wish to create their own software. Presently, Roger works closely with both teachers and developers. He teaches classes to pre-service math teachers and oversees development projects.

Roger has multiple interests in the Math Tools site. First of all, he is interested in broadening the community of math teachers with whom he can become involved. Math Tools will allow him to have discussions with teachers throughout the US and around the world who are interested in technology. Eventually, he would like to get the teachers involved in research, either through discussion of high-level ideas or by systematically giving feedback about technology. He is also interested in connecting with other researchers, as well as discussing collaborations between teachers and developers. Ultimately, he wants to keep his finger on the pulse of both research and practice so he can shape his research questions and technology designs to the most current and important issues.

Towards Integrated Design: The Workshop Metaphor

Primary findings of the user study revealed the need for search and publishing in the DL, the need for community around the DL, and the need to communicate with others about math tools. To support an integrated design process that draws on the these findings, we developed a metaphor and several design principles to organize our ideas, using a meta-architecture approach (Malan & Bredemeyer, 2003). A design metaphor describes a system in concrete terms that can be discussed and refined by both technical and non-technical stakeholders in the design process, without the danger of committing prematurely to design elements in the actual system. A design principle states an organizing idea that enables many related decisions to be made in a coherent way. When combined with the concerns, priorities, and personas, the introduction of metaphors and principles can help to guide the design process.

Organizing Metaphor

The central metaphor we developed for Math Tools is imagining the developers' area as a "workshop," as in a machine shop or tool cooperative. (A real life instance of this is CELLspace in San Francisco, a community arts incubator and learning environment [3].) The workshop metaphor is uniquely suited to Math Tools, which differs from most other digital collections in that many of the resources may be in an unfinished state. The site will serve partially as an incubator for evolving math tools where teachers can help to shape the development of the tools by contributing ideas, feedback from the classroom, and lesson plans.

The workshop metaphor addresses the need for search and publishing by providing a natural place within the Math Tools digital library to look for, share, and contribute to works-in-progress, as well as to consult past work and reference materials.

The metaphor addresses the need for community by organizing the online space into workbenches manned by individual developers yet visible to the walk-in visitor. A workbench may be personalized to create and display one's identity and to serve as an online profile. In addition, Math Tools could provide a "matchmaking" service to connect teachers and developers with complementary needs.

The workshop metaphor supports the need for users to communicate with others about math tools by allowing workbenches to serve as community meeting places by being the focus of events and tool-specific discussions. Finally, while the atmosphere of a workshop is such that everyone is working on individual projects, the work occurs in a public setting that creates a natural focus for conversation and mutual support. This accurately reflects the needs of our interviewees, who expressed a desire for support of their own works-in-progress without overcoming the much higher threshold of first identifying a new project members may want to initiate jointly.

Design Principles

A first principle of design suggested by reviewing the personas was the need to create a space appealing and sensitive to the needs of different people. Different personas may have different aesthetic sensibilities and usability requirements. For example, Drew the Developer may be comfortable rolling up his sleeves and navigating a cluttered workshop, but Tammy the Teacher may need a walkway or guided tour.

Given the range of our different personas from novice (Tammy) to expert (Tom and Drew), another important principle would be designing for a range of roles (Kim, 2000). A workshop typically accommodates various levels of expertise, from apprentice to expert, but the physical layout has an absence of hierarchy. Novices as well as experts may set up workbenches. Experts occasionally roam the floor to view what is going on, mingle with other experts, and mentor less experienced developers. Math Tools may want to develop volunteers or staff in this capacity to encourage participation. During our discussions of the workshop metaphor we introduced an additional persona of "Stan the model shop guy" who was a constant presence and always offered advice whether or not such advice was sought.

Another supporting principle related to the workshop metaphor was the need to have indicators of levels of activity to reinforce the sense of community and overcome isolation. Interaction in the workshop takes place in discussion forums or chat rooms that may be associated with each workbench or with the entire workshop floor. In a physical workshop, the presence of several people clustered around a workbench would indicate that something interesting is happening. In an asynchronous online workshop, some other indicators of activity level are necessary, for example, how many people have downloaded or reviewed a tool, how many discussion postings or chat participants there are, etc. This may draw people to the most active workbenches. Math Tools could attract people to new workbenches, or to well-established but less active workbenches, by announcing milestones or organizing events around the launch of a workbench or the release of a new tool.

Conclusion

We have found that a formative user study can lead to insights that support the design of an NSDL collection as a social place, reflecting the view that "libraries are meeting places where joint research is carried out; research is a highly collaborative activity, even if this is not officially acknowledged" (Levy & Marshal, 1995, p. 80). While publishing and search are important activities, users also want to overcome isolation and to engage others in supportive talk about their own projects. The personas we created cast these concerns in memorable form, and provide imaginary characters that can be used in solving further design problems. We suggest that a "workshop" metaphor may be appropriate for addressing these concerns with respect to Math Tools DL members who wish to use the library to advance their works-in-progress. Supporting principles included: creating an appealing space for different personas; fostering differentiated roles on a continuum from novice to expert; and revealing levels of activity to draw members to lively areas.

As with any metaphor, there are limitations. The workshop metaphor is suitable for members who have projects, but would need to be extended to accommodate those without projects. One extension might be to treat a teacher's goal of "finding a great tool for my Algebra I class" as a project. In addition, the metaphor starts to break when developers upload content to Math Tools, because the work is really performed on the developer's computer, not in the public workshop. Still, the metaphor contextualizes features and interactions for the Math Tools designers in a way that can be referred to, adapted and refined during the design process.

Acknowledgements

We thank Gene Klotz and Steve Weimar, Math Forum, Drexel University, for offering us the opportunity to work on this project in support of their Math Tools collection, and for being such an involved audience for our user study design, findings and recommendations. We also thank Chris DiGiano, Michael Griffin, and Anders Rosenquist at SRI International for their important input during this project. This work was supported by National Science Foundation grant #0226284, The NSDL Math Tools project.

Notes

[1] The following quote describes libraries as shared workplaces: "...libraries serve social and intellectual roles by bringing together people and ideas...allowing people with different perspectives to interact in a knowledge space that is both larger and more general than any single discipline or affinity group" (Marchionini & Maurer, 1995, p. 68).

[2] The Math Tools project (http://mathforum.org/mathtools/) is part of Drexel University's Math Forum (http://www.mathforum.org).

[3]The physical space of CELLspace is described on the CELLspace web site at <http://www.cellspace.org>.

References

Borgman, C. (1996). Social aspects of digital libraries. Los Angeles: UCLA.

Borgman, C. (1999). "What are digital libraries? Competing visions." Information Processing and Management, 35(3), 227-243.

Cooper, A., & Reimann, R. (2003). About face 2.0: The essentials of interaction design. New York: John Wiley & Sons.

Kim, A. J. (2000). Community building on the web. Berkeley, CA: Peachpit Press.

Levy, D. M. & C. C. Marshall (1995). "Going digital: a look at assumptions underlying digital libraries." Communications of the ACM, 38(4): 77-84.

Malan, R., & Bredemeyer, D. (2003). Software architecture: Central concerns, key decisions. Bredemeyer Consulting. Retrieved November 18, 2003, from the World Wide Web: <http://www.bredemeyer.com/pdf_files/ArchitectureDefinition.PDF>.

Marchionini, G., & Maurer, H. (1995). "The roles of digital libraries in teaching and learning." Communications of the ACM, 38(4), 67-75.

National Research Council. (1998). Developing a digital national library for undergraduate science, mathematics, engineering, and technology education. Washington DC: National Academy Press.

NSDL. (2003). NSDL library architecture: An overview. NSDL. Retrieved November 18, 2003, from the World Wide Web: <http://comm.nsdl.org/download.php/293/NSDLArchOverview-2003.doc>.

Project i-DLR. (2003). Usability and evaluation research. Interactive Digital Library Resource Information System. Retrieved November 18, 2003, from the World Wide Web:<http://tiger.coe.missouri.edu/~rafee/iDLR/viewpaper.php?pid=24 >.

Roschelle, J., DiGiano, C., Chung, M., Repenning, A., Tager, S., & Treinen, M. (2000, December 12-15, 2000). "Reusability and interoperability of tools for mathematics learning: Lessons from the escot project." Paper presented at the International ICSC Congress on Intelligent Systems & Applications: Symposium on Interactive and Collaborative Computing, Wollongong, Australia.

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | In Brief
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/february2004-shechtman