60.1 COMMENTS ON UBELL SURVEY (See issue no. 59)
Various subscribers.
>From Ann Okerson, Association of Research Libraries, ann@cni.org (first posted on COLLDV-L; used with permission):
.... Robert Ubell is consultant to the publishing industry. He and his staff take on contract projects for clients. I don't remember how long ago -- possibly early in 1992 -- I had a phone call from them saying that they were going to put together a study of the prognosis for the academic/scholarly/professional budgets and buying through the 90s (forgive any inaccuracies or vagueness; it was quite a while ago). This was to give important segments of the industry some guidance as to market planning. The Ubell company wanted to know if we would like to be a client, since it was our (type of) market that would be under study and the information might be of use to us as well as to publishers and vendors and distributors. For a $15,000 fee, the client could help shape the study, its questions, its methodology (to some extent, although Ubell are very good at this), influence the way the focus groups would be done, and so on. And then share in the results.
We consulted in the office and replied that we felt we should not participate as clients. In part, the ARL is not budgeted for such activities; also, the ARL statistics and the member libraries would presumably be some of the primary sources for the trend and market data gathered, and even if we did participate it seemed to make sense that it be more of a consultant role in which we did not pay out money. Also, our time is tight and priorities are set by the Board and by our members, and this was not necessarily in scope. So we respectfully declined and while we did send them a list of our members (this is a matter of public record anyhow), we asked that as per ARL policy, it be clear that neither the communications we'd had, nor the list, could be construed as any endorsement of the survey by the Association. This latter in large part because the members are surveyed constantly; the surveys take a great deal of time to complete; and time is precious. ARL surveys may be more likely to be completed than some others and we don't want to burden members with any more than is already on the library plates.
I met Bob Ubell at a meeting in late September and inquired about the project, which he said was going ahead. He also said that some focus groups were about to happen or had happened, and that a small but prestigious list of clients had been assembled. I don't recall the names precisely, but they did include at least one major subscription agency, a couple of learned scientific societies, and some large publishers. Again apologies for the imprecision -- my memory for who they were, specifically, is mostly nil.
So, the bottom line: Ubell Associates are not fronting for G & B. (They have never done so, to our knowledge, in the earlier survey(s)). They are a reputable organization undertaking a serious market study to help publishers and distributors plan their programs and products and futures. Any decision to answer the questions or not, then, should be undertaken on the more pragmatic issues of: How much time does one have to devote to a such a study for the industry? Is educating and informing the industry a priority in the organization?
Several librarians have discussed this (with me) over the past few months as we get more and more calls from consultants, venture capital companies, and others not directly engaged with libraries. One prevalent feeling is that time might be better spent where the library gets a more direct return.
Sorry to clutter screens about this. The description of events and the views are mine rather than my employers' or the members of the ARL....
>From Peter Graham, Rutgers University Library, GRAHAM@ZODIAC.BITNET:
I continue to regard the questionnaire as suspicious. The outstanding reason is that it is a publisher's questionnaire, not a disinterested one; as Marcia pointed out, one of the putatively significant questions about journal consideration did not include "pricing" as an answer alternative.
The design of the questionnaire and its distribution seem to me very open to question from a methodological point of view, and I speak from some experience in an earlier career working on just such matters. Sending out 18,000 questionnaires is a major study, make no mistake about it; there is significant money being invested in the production, mailing, coding, data entry, data cleaning, analysis and use of this survey.
Sending out a study to 18,000 individual respondents when the questions in many cases are directed at institutional behavior raises a methodological issue as to whether the results can be of real use. Let us say 80% of the 50 librarians at one institution answer the questionnaire correctly about their institution's fiscal stance and behavior, and 1% of another institution's do. What meaning does the raw count of responses provide?
Also at the most general level: if a study is being done of serials issues, surely the community of interest should include others than those who produce them. In fact it appears that this may be a problem we should not blame on the publishers but on an overzealous market research firm who does not see the need to consider the larger issues before doing the survey (much less doing a pre-test survey and looking at the results with informed parties).
Marcia Tuttle kindly says that the study may be "legitimate" and that libraries and patrons may "benefit as well" even though the study is being done for publishers. To my mind, the study is only legitimate in the sense that we don't want to censor anyone; it does not seem a well-designed study either in intent or methodology, and I will let a more informed expert determine its legitimacy in that sense. And I disagree that we will benefit from a poorly-designed study that, among other things, may perhaps have the negative effect, by its existence, of discouraging better constructed studies.
And just what library organization rented an 18,000-name list to a market research firm without asking what was going to be done with it? And how did so many of us who are not sci-tech librarians or serials purchasers get the form?
From: Peter Stangl, Stanford University Medical Center Library, peter@krypton.Stanford.EDU:
Thanks for the sober piece on the Ubell survey in #59. I am no bleeding heart fan of publishers in the currently raging confrontation, but find the paranoia displayed by so many of my learned colleagues quite tiresome. As you quote Chet Grycz, there are no simple or single solutions -- what we need most of all are open minds, clear vision, creative judgment, and good faith. Fanatical, a-priori condemnation of anything and everything a for-profit publisher says or does shuts out any chance of progress. In fact, approaching our funding crisis with complete disregard for the publishers' interests in this, our capitalistic system of government, is just as obstructionist or counter-productive as the strategy we criticize in the "enemy" -- the publisher...
>From James Mouw, University of Chicago Library, mouw@midway.uchicago.edu:
I was called about possibly participating in the Ubell Chicago Focus group. I spent maybe 15 minutes on the telephone answering general questions about our collections (nothing as specific as is on the questionaire). I was not selected for the focus group. It would be interesting to hear from anyone who DID attend one of the focus groups.
>From Kate McCain, Drexel University School of Information Studies, MCCAINKW@DUVM.OCS.DREXEL.EDU:
Your characterization of the survey suggests just how difficult it is for "consultants" to get in touch with the _real_ issues when they dive in without a lot of real knowledge or expertise. I would be interested in knowing -- for instance -- just how representative any "focus groups" or individuals were who they used to pretest the questionnaire -- a necessary step in any competent survey research.
And this follow-up from Peter Graham:
Today (Friday) I was able to speak with Mr. Ubell and Mark Tesoriero, as we were all at the Coalition for Networked Information meeting. (Ubell and Associates, it should be noted, have been contracted by CNI to do certain work.) I introduced myself so as not to hide behind the net and they very graciously discussed some of my concerns.
On methodology: my concern about inferring institutional behavior from individual reponses they address by saying that the crucial question on the form is the last one which asks respondents to identify their role in the library. They will exclude from their analysis persons who do not report a sci-tech budgetary responsibility, as I understand it. Thus if you are a humanities person at a college you will be excluded; if you are the head of collection development you will be included; etc. Their mailing list was intended to comprise sci-tech people of this kind but they know it won't be pure; others will get it (and will be excluded).
I asked why they didn't ask for the information they want from the libraries instead of this roundabout method. They professed not to know how to get this information; that is, that they find it impossible to determine who in a library is the responsible person to address such a questionnaire to.
They said their focus groups had given them a high degree of confidence that they knew what the survey would come up with; that is, Mr Ubell expected that he could say now within 3-4% what library activities being asked about were going to be in the coming period (e.g. some but not much more document delivery to go on, etc.). I said that given a day or two I could probably do the same within about 10% accuracy. He agreed, and said that from his clients' point of view the issue was that they didn't have that kind of knowledge.
I asked about the kinds of people in the focus groups. All kinds of librarians, he said. Library managers, I asked? Well, no, he said.
I expressed in parting that my concerns arose out of a sense, justified or not, that an end run on the library community was being done. He disagreed. I tried to get across that I was not alone in this concern and that he might have done well for his own purposes to address this in the cover material for the survey; I don't believe he agreed that this was important.
I added that I saw some danger that the clients, whoever they are, might misread the data as representing libraries in a way that I wasn't sure was justified; and that it was a value to me to be sure that the vendors got good information, even if we disagree. Ubell did note that the results of the study will not be published -- we'll never see them.
I continue to believe that the survey is inappropriate as a means of determining what libraries are doing and planning, and inappropriate as a means of maintaining a sense of collegiality and community between those who speak for publishers and those who speak for libraries. Thus on the one hand I have doubts about the quality of the information being gathered. And on the other hand the process itself, when considered with such behaviors as publisher lawsuits, seems confrontational rather than cooperative. A colleague of mine here at Rutgers unprompted called the survey "intrusive"; the blindness to that perception is a continuing publisher problem.
>From Paul Gherman, Virginia Tech, GHERMAN@VTVM1.BITNET:
I would like to add a note about Ubell and Associates. CNI hired them to conduct focus group sessions last summer to see if librarians and publishers could reach agreement on the READI Program which was proposed by CNI. I attended the session for librarians, and felt that Robert Ubell and Associates were a very professional group and conducted a very well run session. I cannot speak for their clients, or the study, but the firm is top rate.