D-Lib Magazine
October 2005

Volume 11 Number 10

ISSN 1082-9873

The CREE Project

Investigating User Requirements for Searching within Institutional Environments

Chris Awre*, Gabriel Hanganu†, Caroline Ingram‡, Tony Brett†, and Ian Dolphin*

*e-Services Integration Group University of Hull
Hull, UK
{c.awre, i.dolphin}@hull.ac.uk>

†Oxford University Computing Services Oxford, UK
gabriel.hanganu@exeter.ox.ac.uk
tony.brett@oucs.ox.ac.uk

‡42 Coquet Terrace
Newcastle upon Tyne, UK
caroline@csiconsultancy.co.uk

Abstract

The Contextual Resource Evaluation Environment (CREE) project has investigated user requirements for the presentation of a range of different Internet-based search tools in a variety of local institutional environments, enabling access to the search tools away from their home websites. The project's three-strand approach to its investigation included a survey, focus groups and user testing. The investigation revealed that there is much interest in the idea of using Internet search tools through institutional environments, although this interest is tinged with some uncertainty on the part of end-users about the exact purpose or need for presenting search tools this way. Presentation of search tools through institutional environments does, though, increase end-user awareness of what search tools are available and potentially draws them into using new resources.

Introduction

Under the aegis of the UK Joint Information Systems Committee's (JISC) Portals Programme [1], development projects have taken place in the UK that investigate the use of portals as presentation paths for a variety of search tools. A major output from these projects has been the development of portal interfaces, websites that users can come to in order to make use of the functionality that these portals provide, particularly searching. Each project, as a key part of its investigations, created their own such interface and website. These were tested with end-users to establish that their portal was presenting its services in the most useful manner. The work the JISC portal projects have carried out on portal interfaces has been valuable in understanding better the types of services with which end-users wish to interact and where they perceive value to sit within the portals.

Towards the end of the portal projects it became apparent, though, that the presentation of services through a dedicated website was just one way in which services could be delivered. Many institutions are now making use of virtual learning environments/course management systems (VLE/CMS), and a number of institutions are starting to implement institutional portals to facilitate the aggregation and presentation of applications, services and information to their staff and students [2]. All universities also work heavily within the general web environment, providing a vast collection of information to those both inside and outside the institution. These institutional environments have made it possible to bring information and services to end-users in the context of their work and/or study. This delivery of information and services, including search, to the end-user contrasts with the more traditional approach of building dedicated websites and expecting or requiring the end-user to find and come to these.

Whilst the provision of dedicated websites is still the predominant route for the delivery of Internet-based search services, JISC considered it important to examine more closely the integration of Internet-based search tools within institutional environments to establish both how this might take place and, in particular, what requirements end-users might have for this re-focussing of where they searched. The CREE project was funded by JISC to investigate this.

The CREE (Contextual Resource Evaluation Environment) project

The CREE project [3] has been structured around two main goals:

1. Investigating user requirements for the presentation and delivery of search tools through a variety of institutional environments and contexts. This work was undertaken by the University of Hull in conjunction with colleagues at the University of Oxford and Newark and Sherwood College (a community college). The cross-section of institutions allowed variances to be identified between institutions and different end-user groups, but also validation of results in common across the three taking part.
2. Investigating the JSR 168 [4] and WSRP [5] portlet standards to allow the presentation of search tools within conformant portal frameworks (e.g., uPortal [6]). This work was undertaken primarily by project technical partners at the University of Oxford, the Archaeology Data Service at the University of York, the EDINA National Data Centre at the University of Edinburgh, and instructional media + magic, inc., a software consultancy in Washington D.C. Each of the partners investigated the use of these standards with a different type of search tool.

This article focuses primarily on the user requirements work undertaken by the CREE project. Technical aspects of the project are reported in a companion paper in Ariadne [7]. The two strands came together in the development of a series of fully functional interactive demonstrators, one of which was based around uPortal and that made use of the portlets developed by the technical partners.

User requirements

The gathering of user requirements has played a large part in influencing and guiding the planning and development of services at the University of Hull. The introduction of an institutional portal in September 2003 [8] was based on the combination of gathering user requirements via surveys (reported through the PORTAL project [9]) and holding meetings with end-users to get feedback from them on what they would like to see. This has been particularly important to help avoid wasteful development effort being taken up by ideas that, whilst seemingly impressive, do not match what end-users would actually like to have.

Gathering user requirements within CREE was similarly seen as important to establish where development effort should be focused in the presentation of search tools within the different institutional environments available. Although it might be nice to present access to a particular resource within a VLE/CMS, would doing so be useful to the end-users of the VLE/CMS? While we acknowledge that these requirements may change with time, establishing a baseline from which to build is valuable in the development of the environments in question.

Methodology

Three complementary approaches to gathering user requirements within CREE were used. They were carried out sequentially, and each approach was used to validate the results of the one before. In this way, results from one approach did not exist in isolation, but were both expanded upon and validated by subsequent activities (Fig. 1).

Fig 1. User requirements gathering process

Survey

The project commenced with an online survey carried out nationally across the UK. This survey provided baseline data upon which the project could build. It sought information primarily in two areas: how and why end-users currently searched for information in general; and what opinions did they have on the possibility of searching from within another environment. (The VLE/CMS option was used here, as this environment is now relatively common amongst UK universities and was therefore familiar to survey respondents.) The survey made use in part of an innovative drag and drop card-sort toolkit that facilitated the usability of the survey and encouraged a good response. As a further incentive a prize draw for an iPod was offered to respondents. In return we were fortunate to receive over 2000 responses, providing a valuable and reliable set of results for subsequent use.

Focus groups

The survey was followed by a number of focus groups held across the partner sites involved in the user requirements gathering, and these groups addressed the same issues as the survey. The focus groups offered two advantages: they allowed the results from the survey to be tested and validated; and they allowed this testing to take place in a group environment. Surveys offer responses based on individual views: the focus groups offered the opportunity to bounce these individual views around and provide either a group view or an agreed difference in views between individuals. The reasons for both could also be examined, providing further information on why certain views were held. Amazon vouchers and entry into an iPod draw were provided as an incentive, attracting over 60 people to a series of 13 focus groups across the partner sites.

User testing

Both the survey and the focus groups had gathered user requirements based on theoretical opinions and views held, particularly in relation to the use of search tools in different contexts. Very few of the end-users involved had previous experience using search tools (or other functionality) in such environments. Three fully functional interactive demonstrators containing a number of integrated search boxes were created to allow users to play and experiment with what it might be like to use search tools in different contexts and offer feedback via a questionnaire on the basis of use rather than asking for speculation. The demonstrators offered access to a range of different search tool types:

• Library catalogue (University of Hull [10])
• Internet search engine (Google [11])
• Bibliographic resource (zetoc [12] or GetRef [13])
• Subject-specific resource (search tools from the Archaeology Data Service [14])
• Reference resource (Dictionary.com [15])
• Websites resource (the Resource Discovery Network [16])

Participants in the user testing were offered vouchers as well as entry into an iPod draw as an incentive. Over 70 people took part across the three partner sites.

The demonstrators covered three institutional environments that either exist already or are increasingly being implemented across universities in the UK.

• Web environment
• VLE/CMS environment
• Institutional portal environment

The web environment has existed within universities for many years. From a Library's perspective this web environment has allowed the provision of access to a wide range of electronic resources, many of them based around the ability to search for information. This provision has, though, largely been limited to a series of links to dedicated, home websites for these services, immediately leading the end-user away from the Library website. One exception to this has been the ability to search the library catalogue from the front page of the Library website (e.g., University of Hull [17]). How might extending this to other search tools, making them more directly available within the local web environment, be regarded?

The VLE/CMS environment is rapidly becoming an established piece of institutional infrastructure for the support of learning and teaching, and the majority of Higher and Further Education institutions in the UK have a VLE/CMS environment in place. The connection between the Library and the VLE/CMS was examined [18] and tested through a series of projects within the JISC's DiVLE Programme [19] in 2003, and a number of such connections were put in place. Reading list software (e.g., Sentient Discover [20]) also allows the connection between library and e-learning systems to be built. The CREE VLE/CMS demonstrator (a mocked-up web page in this instance, as available VLE/CMS systems did not permit the required level of integration) sought to identify how users responded to the presentation of search tools in this environment.

Many institutions have and are examining the use of institutional portals to support the business of the university. This emerging institutional portal environment has the capability of bringing together, in one place, all the processes and activities of the institution. The Library is no exception, and this demonstrator, built on uPortal and testing out the JSR 168 and WSRP standards examined in CREE's technical strand, sought to provide guidance on the positioning of Library services within this portal environment. Whereas the majority of the integrated search boxes acted as direct links back to the home website of the search tool (shallow integration), the portlets offered a deeper level of integration by presenting the whole search process within the portal.

Screenshots of each demonstrator can be found in Appendix A.

Survey results

The results from the survey can be split into two halves, covering the two main areas: how end-users search; and end-user opinions on searching.

In considering the existing use of search tools the following figures emerged:

• 85% used Internet search tools on a daily basis

• 61% used them daily for learning, teaching or research

• 67% used subject-specific resources
• 68% used their local library catalogue at least weekly

• 54% used other library catalogues at least monthly

The high use of search tools is not, perhaps, surprising, and responses to one of the survey questions made it clear that the vast majority of respondents used Google. But the use of more subject-specific search tools was also high. And whereas a figure of two-thirds of users using their local library catalogue at least weekly is not that surprising, the fact that over half of users also made use of other library catalogues at least monthly is notable.

When asked about the use of search tools whilst also using a VLE/CMS (Fig. 2), usage was reasonable, though not high, and about a third never searched at the same time as using the VLE/CMS. Internet search tools in the survey were taken to mean any type of search tool available via the World Wide Web.

Fig 2. When using online learning environments, how often do you use ...?

The survey then moved on to examine views and opinions on potentially different ways in which search tools could be presented. When asked about the potential of searching from within a VLE/CMS, participants demonstrated a sense of uncertainty about what this might offer and opted for caution (Fig. 3).

Fig 3. Would you find it useful to conduct a search of any resource from within an online learning environment?

Only 10% of respondents said they would never use search tools from within a VLE/CMS, however, revealing users' interest to using the search tools within a VLE/CMS dependent on what the tools would offer the task at hand. There was greater certainty among survey respondents about the ability to receive search results within the VLE, as it was the results that could then be used in the context of other learning and teaching activities (Fig. 4). This enthusiastic response was, nevertheless, almost matched by those who were again uncertain about the usefulness of this functionality.

Fig 4. Would you like the results of your search to appear within your VLE session?

The survey also provided baseline information in the following areas:

• 16% of respondents always made use of advanced search facilities where these were available. The majority (53%) used them dependent on the search being carried out, though 27% also limited their use of advanced search to the specific search tool being used.
• The ranking associated with the display of results should be, not surprisingly, by relevance (97% considered this to be important or very important) or by subject (70%). Ranking by date, a common feature in many existing search tools to allow the presentation of more recent articles/books, etc. at the top of the list, received the backing of only 33%.

Another major theme explored by the survey was cross-searching or searching of more than one resource at the same time (Fig. 5). The results again showed end-user caution (though it is assumed that at least some respondents would have access to such a facility due to the uptake of metasearch products within UK universities), with users offering greater support for searching across more than one library catalogue, but not so much support for searching library catalogues and the Internet together. Very few participants indicated they would never like to carry out cross-searching though.

Fig 5. Fig 5. Would you find it useful to search more than one ... at once?

Focus group results

The focus groups allowed the bare facts, outlined above, of how end-users searched to be explored further. Understanding why and how end-users search can inform the presentation of search tools and facilitate the search process.

There was clear acknowledgement across the focus groups of the superiority of Google as a first port of call for searching, though one participant did question whether this was just because he assumed it was better than other search tools, because of reputation and his previous use of Google, rather than by his regularly assessing Google to see if it was indeed superior. Notwithstanding their initial approach to searching using Google, participants were, on the whole, very aware of other resources available to them, and they did use these other resources according to need. However, most tended to stick to the resources they already knew and were reluctant to spend time investigating and trying out new resources and search tools unless it was clear how they would benefit from using them. Human recommendation (via friends, colleagues, library sessions, etc.) was much valued as a prompt towards learning about a new search tool.

The use of search tools was guided by two main search strategies:

• For the completion of a piece of work (this may be an assignment, developing a lecture or writing a report), finding relevant information from whatever source was imperative. Thus, Google tended to be preferred, by both staff and students, as the quickest way of getting something to work with. Searching Google also provided leads that took end-users into more subject-specific and structured resources, and it is was at this stage that end-users would then make use of these additional resources.
• For research purposes, a more considered approach to searching was used. The initial search via Google, when nothing was known about the subject being researched, was still used, but many researchers, chose to focus their searches on known databases and to take their time doing this. The key factor here was their having the time in which to do this type of searching.

It was noted by a number of participants that successful initial use of a search tool would frequently lead to further and more advanced use of that tool. This applied to Google as much as to any other search tools, but it also pointed to the need to make search tools easier to use, drawing the end-user in and encouraging them to make use of the functionality available.

Another distinction in search strategy was identified according to the stage of the search. The majority of searches undertaken were to discover information, particularly if the subject was new and a starting point was required. Once information was discovered, it needed to be located, and there was a clear end-user request for the link between these two to be as seamless as possible. It was here that the library catalogue was mentioned most frequently: the catalogue was rarely used to discover information about books, for which the Amazon catalogue was favoured by many participants, but the catalogue was used simply to see if the item was available locally. The library catalogue was used primarily as a source of location for books, while other resources were used more frequently for journals (such as e-journal listings). For e-journals, end-users displayed a marked preference for retrieving only those items that were available in full-text. As with one of the reasons given for using Google, time was given as a factor driving this.

With regard to using search tools whilst connected to a VLE/CMS or an institutional portal, the focus groups reflected the survey finding that this did take place, but not to any great extent. Although focus group participants considered it a possibly useful idea to present search tools within the VLE/CMS or portal, many participants, particularly those who were students, preferred their current practice of keeping a number of separate windows open and flicking back and forth between them to access whichever service they needed. It was notable, though, that those participants who were frequent and regular users of a VLE/CMS or portal were much more open to the idea of having search tools included in the VLE/CMS or portal: such users saw search as an additional piece of functionality that would complement what was already available; those not familiar with the VLE/CMS or portal did not see the value of integrating search somewhere else or of the extra step required to access a particular search tool if it was presented via this route. If search tools were to be presented within the VLE/CMS or portal, it was thought that quick reference tools such as dictionaries or thesauri would be most useful.

On the issue of cross-searching, the focus groups exhibited the same caution as was indicated by the survey respondents. Almost all thought cross-searching a good idea, especially if it saved time over searching resources one by one (and staff were more in favour in this respect than were students). But there was concern about how the results of such searches would be displayed. The favoured option was for each result set to be displayed separately so that end-users could clearly see which results had come from where. A preference was expressed for only cross-searching resources of the same type (e.g., library catalogues or bibliographic databases, but not both), and the issue of results display clearly influenced this preference. In summary, it was considered that cross-searching and particularly the display of results from this way of searching had to be done well; otherwise the results would be unusable and therefore cross-searching not worth doing.

User testing results

The results from the user testing sessions highlighted the benefit of placing users in front of a system with which they could interact, and gathering feedback from this. Participants were very enthusiastic about the possibilities of presenting search tools in different contexts. One indication of this came from noting the length of time participants took to complete the questionnaire: although it had been estimated that the questionnaire would take 30 minutes to complete, many participants took longer than that to answer the questions, and they indicated that they had become involved in using the demonstrators far more than they had expected.

The method of presenting the search tools in the different demonstrator contexts, integrating search boxes via various techniques, was clearly welcomed by participants, who largely found the tools easy to use. This was particularly the case with the web and VLE/CMS demonstrators, which most closely resembled existing practices. For the most part the search boxes acted as a means to direct the end-user to the home website of the search tool, either in a separate window or in the same window. The Resource Discovery Network 'include' service [21] was used to present search results in a local web page setting and with the same look and feel as an alternative option. The JSR 168 portlets used within the institutional portal demonstrator also kept the user within the local environment, thus removing the need for the user to leave the portal. This difference between the web and VLE/CMS demonstrators and the portal demonstrator was reflected in responses on the ease of use for the portal demonstrator; the responses indicated that the portal demonstrator was more difficult to use, in part due to end-user lack of familiarity with it. The options for displaying results were important to users. The demonstrators offered different options, and these received the following backing across the three demonstrators (Table 1).

The use of a separate window was the favourite, though there was also good support for keeping the search within the local environment.

The main focus of the questionnaire was on whether participants would make use of such search tools if they were presented in this way in a real setting. Table 2 highlights the different responses gained for the different types of search tool used.

It can be seen from these results that the highest end-user preference is for access to bibliographic, subject and reference resources. The latter reflects the opinion expressed in the focus groups that quick reference tools would be the most useful if presented in different environments. Comments made by participants of the user testing sessions made it clear, though, that, having seen the possibilities for presenting different search tool types, it would be most valuable to have access to resources with which they were unfamiliar presented in this way, particularly where the search tools were relevant to an end-user's subject area. Integrating the search tool, and thereby taking it to end-users, was seen as a means for alerting them about resources they might not previously have come across, and doing so encouraged the end-users to try out new options.

The likelihood of end-users searching library catalogues and using Internet search engines within different institutional environments was reasonable, though not particularly high. Comments suggested that this was due to users knowing about these search tools already: 'Why access Google through the VLE/CMS when I'm so used to accessing it elsewhere?' was a frequent sentiment.

It is also notable from the results shown in Table 2 that the VLE/CMS was regarded as the most popular environment in which to present search tools. Participants indicated a connection between searching and the learning and teaching activities carried out through the VLE/CMS system. The presentation of search tools in other environments, i.e., within local web pages or in an institutional portal, was considered valuable by end-users but only when placed in the right context.

When presenting search tools in different environments, the tools can optionally be presented singly, in a group, or alongside other functionality within those environments. As results shown in Tables 3 and 4 indicate, participants favoured the use of groups, and it was clear that they particularly wanted a single place to which they could go that would provide them with access to the search tools they needed. Even though the search tools were being presented in different environments, participants tended to prefer a separation of roles: it was valuable to have the search tools present, but only in their place and not mixed in with other functions.

The user testing also examined the issue of cross-searching and reached very similar conclusions to the survey and focus groups. In principle, end-users welcomed cross-searching, but they wanted it to be clear what the benefit would be to have it available and be clear how cross-searching would be used.

Conclusions

The investigations carried out by the CREE project have shown a clear interest on the part of end-users to make use of search tools when these tools are presented in different local institutional environments. It is equally clear that presenting the tools through these environments would represent a big change in the pattern of usage – a change about which many users are currently uncertain.

The CREE project team had anticipated that presenting search tools in different environments, and thereby associating searching with other areas of functionality, would be valued by end-users. However, results of the CREE study indicate that end-users prefer a separation between searching and other functions, even if these functionalities become more closely associated within the local online environment. The main benefit to emerge from the user testing was the finding that presenting search tools in different environments to end-users was regarded by users as a way of their being alerted to and led to different search tools about which they currently did not either know or use. The added value of bringing search to the end-user within a local environment is thus to increase their awareness of the facilities available to them and to give them a greater chance to see what could be most useful to them. This is clearly a role that libraries can take on, to facilitate and assist end-user access to the broad information landscape that is available.

Further information

All reports from the CREE project are available via the CREE website at <http://www.hull.ac.uk/esig/cree>.

Acknowledgements

The authors would like to thank Gary Thompson and Wayne Thompson in the e-Services Integration Group at the University of Hull for their assistance in creating the online survey and demonstrators, respectively.

References

1. JISC Portals Programme, <http://www.jisc.ac.uk/programme_portals.html>.

2. JISC Study of MLE activity, <http://www.jisc.ac.uk/project_mle_activity.html>.

3. CREE Project website, <http://www.hull.ac.uk/esig/cree/>.

4. Introduction to JSR 168 - the portlet specification, <http://developers.sun.com/prodtech/portalserver/reference/techart/jsr168/>.

5. OASIS Web Services for Remote Portlets Technical Committee, <http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsrp>.

6. uPortal, <http://www.uportal.org/>.

7. Putting the library into the institution: using JSR 168 and WSRP to enable search within portal frameworks, <http://www.ariadne.ac.uk/issue45/awre/> (available from 1st November 2005).

8. University of Hull portal tour, <http://www.hull.ac.uk/esig/portaltour/>.

9. PORTAL project website, <http://www.fair-portal.hull.ac.uk/>.

10. University of Hull Library Catalogue, <http://library.hull.ac.uk/>.

11. Google, <http://www.google.com/>.

12. Zetoc electronic tables of contents, <http://zetoc.mimas.ac.uk/>.

13. GetRef, <http://www.edina.ac.uk/getref/>.

14. Archaeology Data Service, University of York, <http://ads.ahds.ac.uk/>.

15. Dictionary.com, <http://www.dictionary.com/>.

16. Resource Discovery Network, <http://www.rdn.ac.uk/>.

17. University of Hull Library website, <http://www.hull.ac.uk/lib/>.

18. INSPIRAL Project website, <http://inspiral.cdlr.strath.ac.uk/>.

19. JISC Linking Digital Libraries with VLEs (DiVLE) Programme, <http://www.jisc.ac.uk/programme_divle.html>.

20. Sentient Discover, <http://www.sentientlearning.com/home/Solutions/Sentient+DISCOVER/>.

21. RDN-Include service, <http://www.rdn.ac.uk/rdn-i/>.

Appendix A - Demonstrator screenshots

Local web page environment demonstrator

VLE/CMS environment demonstrator

Institutional portal environment demonstrator

Top | Contents
Search | Author Index | Title Index | Back Issues
Editorial | Next article
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/october2005-awre