by Charles R. McClure, John Carlo Bertot and Carol A. Hert
The 1999 ASIS Mid-Year Meeting (MYM) in Pasadena
(May 24-26) presents an outstanding opportunity to expand our knowledge of evaluating networked information services and resources. As co-chairs of the meeting, we would like to provide a preview of some of the important and
timely issues that the Society will be discussing there.
Access to and use of networked information resources and services over the Internet continue to explode. Traditional uses, such as e-mail, Web searching and
uploading/downloading files, have continued to grow exponentially at the same time that new and innovative interactive services and applications – such as moving from "Webs" to "portals" – are also appearing and taking hold at
an astonishing rate. Commercial applications and uses of the network also are growing. As the "Digital Economy" expands, multiple impacts on society will occur.
networked information services and resources
designates many complicated interactions among technology, policy, organizations and people that require ongoing thought and clarification. Network-based resources and services include those electronic information resources and/or services that users access from their homes, workplaces, libraries or other locations via public, commercial, statewide or national networks. Some examples are Web sites providing public and commercial information, as well as library-licensed databases (e.g., Infotrac, SearchBank, EbscoHost). Examples of electronic networked services include e-mail, listservs, chat, online reference/assistance, downloading information or forms, interactive video and "push" technologies that provide users with a range of customized information and services on a regular basis.
This complicated web of technology, services and applications, users and organizations and the need to assess its implications and impact led to the planning of this Mid-Year Meeting. The call for participation, which can be
outlines the goals and topics that the MYM papers and panels will address. Notions of evaluation and use of networked information services and resources include the
- social implications
As the MYM has evolved, however, a number of key themes that cut across these topics have appeared. This article introduces four of them with the hope of stimulating discussion at the meeting. They are the importance of
- commitment to ongoing network evaluation;
- frameworks for developing evaluation measures;
- user-based evaluation; and
- the impact of information policy.
Commitment to Ongoing Network Evaluation
A typical comment heard from many providers of networked information resources and services is that "there is no time" to engage in ongoing assessments –
regardless of the type. Thus, to some degree, networked information resources and services evaluation is a method in search of a practice. There appears to be much more talk about the use and importance of evaluating networked
information services and resources than actual regular ongoing practice. Often excuses for not doing evaluations seem better developed than ongoing strategies to implement them.
Ongoing evaluation requires time,
resources and knowledgeable staff, but perhaps most importantly, it requires a commitment – usually an administrative commitment – to better understand and improve the services and resources provided via the network. Among
other things, commitment must include a previously agreed upon budget that supports ongoing (not
a one-shot-only effort) evaluation and assessment processes. A general rule of thumb is that at least five to seven percent of a project budget should be dedicated to ongoing (formative) and final or summative evaluation and assessment. How can such commitment to improvement be instilled in providers and developers of networked information resources and services? As we speak, a dedicated line in the project budget for evaluation occurs rarely at best and not at all at worst.
And there are still many questions that evaluators must address. What models for evaluation efforts can be proposed that may offer the best "return on investment" for the resources committed to such evaluations? What types
of evaluation can best improve specific aspects of networked information resources and services? What arguments as to the benefit(s) of evaluating networked information services and resources will move administrators/decision
makers to conduct regular and on-going evaluation efforts?
Frameworks for Developing Measures
Measures that describe networked information services and resources can be complex. Based on a number of
completed studies, the authors, for instance, have suggested one approach. The figure below shows our framework. Space does not allow detailed discussion, but basically the framework suggests that there are a number of network
components, such as technology or management support, that can be assessed in terms of a number of different criteria, including efficiency, effectiveness and usefulness. Two of the authors of this article, Bertot and McClure,
provide a more detailed discussion of this framework in "Measuring Electronic Services in public Libraries: Issues and Recommendations" (Public Libraries, 37[3: 176-180).
While this measurement approach may not be
appropriate for all types of assessments of networked information services and resources, it is a beginning point for analyzing which components of the network can be assessed in terms of which criteria. Moreover, these
approaches may take either the formative or summative strategies mentioned above. At the meeting we hope to explore other frameworks, models or conceptual approaches evaluators can develop for assessing networked information
resources and services. How can such models capture the complex and dynamic nature of these networks?
There also appears to be an evolving discussion on the role of user-based evaluations of
networked information resources and services as opposed to system-based, technical evaluations. A number of the papers and panels for the meeting address this issue head on. A user perspective should consider the culture of the
communities and sub-communities involved, the relationship between community norms and the use of networks, and the individual needs and uses of the network by the users.
A user perspective assumes that technical criteria
alone should not drive the design and implementation of information technologies (IT). Rather, IT design and implementation should take into account the particular communication behavior, information use patterns and work
environments of both users and potential users. Since networks, ultimately, are designed to meet user information needs, it seems obvious that evaluations should contain a user-based component.
User-based approaches for
evaluating networked information services and resources are critical if we are to determine the degree to which those services meet user needs and actually assist users to be more productive in a variety of contexts. They are
essential for successful planning and development of networks and the provision of networked information resources and services. How can we better integrate technical and user-based evaluations? Which types of user-based
evaluations can best inform which types of technical assessments and vice versa? Perhaps most importantly, how can network designers learn from users and better evolve services to meet user needs?
Impact of Information Policy
National information policies have an ongoing and pervasive impact on the development of and access to networked information services and resources. Increasingly, a range of information policy
topics, such as privacy, intellectual property rights, security, records management, access, transborder data flows, copyright and tariffs, have direct impacts on the types of technologies and services that Internet Service
Providers (ISPs) can provide. Issues related to competition in different communications markets as a result of the Telecommunications Act of 1996 (PL 104-104) also have direct impacts on provision of networked services and
Oftentimes, assessments of networked information resources and services fail to consider the national (or state) information policy system and how that context affects the delivery of networked services and
resources. Evaluators may not be aware of some of these policy guidelines, nor might they be aware of the degree to which these guidelines have a direct effect on services provided by the ISP. Developing evaluation methods that
integrate a range of perspectives, such as a policy perspective, for example, appear necessary to obtain "the big picture" in assessing networked services and resources. To what degree do network assessments take into
consideration this broader information policy context? How can assessments of networked information resources and services better consider such information policies?
Expanding the Knowledge and
Clearly there are other key themes than the four identified here to discuss at the meeting. But these themes do suggest the importance of developing and implementing evaluation techniques for networked
information resources and services that rely on a multi-dimensional perspective. Due to the complex technological nature of networks, user behaviors associated with network use and access, and a range of organizational and
external factors (such as information and telecommunications policy) that are constantly changing, such an approach is essential. Perhaps due to this complexity, the reality is that hard evidence of impact or even of
methodologies that can demonstrate the "success" of providing information resources and services in a networked environment are difficult to find.
If the evaluation of networked information resources and services
is to be improved, there is a need to expand both
the knowledge underlying evaluations and the practice of conducting them. Evaluation researchers must leave the confines of their offices and the trappings of academia; practitioners must recognize the need for and techniques by which to conduct successful assessments of networked information resources and services. Finally, both groups then need the support of administrators to conduct such evaluations and begin an iterative process whereby evaluations improve the overall quality and success of networked information resources and services.
The papers and panels at the ASIS Mid-Year Meeting will continue and extend both our knowledge and our practice of understanding networked information resources and services. The extended abstracts for the meeting are