EDITOR’S SUMMARY

Controlled vocabularies or languages are widely used in the field of information retrieval, but questions of whether they are necessary or more effective than free text searching have been unanswered. With the widespread use of free text searching in popular search engines, many in the information retrieval field have wondered how to make controlled vocabularies more useful or effective. Recent studies on this subject have revealed that while controlled vocabularies can produce more precise search results, user characteristics can potentially match those results in the right environment. These studies included a multitude of users from different backgrounds with different search environments. Some of the most important findings of these studies demonstrate that the skill of users and indexers can affect search outcomes just as much as controlled vocabularies.

KEYWORDS

controlled vocabularies
query formulation
search behavior
user studies
information retrieval
subject indexing
cognitive perception


FEATURE

University Metadata and Retrieval:
The Death of the Library Catalog?

by Ying-Hsang Liu

Can we yet say that benefit received is commensurate with the effort of construction? … Even if controlled retrieval language and thesauri are useful, is their uncontrolled proliferation equally useful? ([1], pp. 136-137)

Controlled retrieval languages such as the Medical Subject Headings (MeSH) and the Library of Congress Subject Headings (LCSH) are standard tools for information access in library collections. The debate over the usefulness of these indexing languages first emerged in the 1970s in the field of information retrieval (IR) when scholars such as Salton and Sparck Jones suggested that free text searching was as effective as controlled retrieval languages. More recently, this question has been raised for major American libraries in a report prepared for the Library of Congress, suggesting that relative to automatic subject indexing, manual indexing is not cost-effective [2].

Given users’ widespread comfort and familiarity with free text search in modern search engines, the question of how (if possible) to make controlled vocabularies more useful for searching is significant. In one of the most comprehensive reviews of this issue, Svenonius [3] identified important factors affecting the usefulness of controlled vocabularies, including the nature of the vocabulary, subject discipline, IR system, the skill of indexers and searchers and user requirements. These issues are complex partly because different research communities, such as those dedicated to system-oriented and user-oriented IR research and to knowledge organization, have investigated them separately from different perspectives, with little or no collaborative effort over the years. Further, the cost and value associated with the construction and maintenance of controlled vocabularies also need to be considered in practical settings.

From the IR perspective, the evaluation of IR systems has been concerned with the system effectiveness and user requirements. Most system-oriented IR has been more interested in how to obtain the best ranked search results, whereas user-oriented IR research has focused on user characteristics, user-system interactions and user search behavior. To evaluate the usefulness of controlled vocabularies in support of search tasks, we need to specifically consider the effect of controlled vocabularies and individual differences of users and search systems on search performance. Empirical studies conducted by Salton and those reviewed by Sparck Jones, regarding the effectiveness of controlled indexing languages, were mostly conducted within a laboratory environment, without the involvement of users in a search environment, similar to modern search engines. One may argue that controlled retrieval languages are no longer necessary, but many operational IR systems, particularly online databases, are equipped with manually created subject index terms.

In this short article, we will discuss studies assessing

  • The effectiveness of using thesaurus terms to supplement the sparse metadata records of a research repository from a system-oriented IR perspective [4].
  • The impact of user search experience on search performance in controlled settings regardless of IR system differences, using meta-analysis [5].
  • The potential search effectiveness of MeSH terms in an interactive search environment from a controlled IR user experiment when used by different types of searchers [6].

Finally, we will discuss a recent eye-tracking user study [7, 8] that was designed to help build natural user search interfaces to support biomedical domain experts in formulating complex queries using MeSH terms.

In the study by Hider et al. [4], thesaurus terms were used to supplement the sparse bibliographic records of a research repository in the education domain. A system-oriented IR experiment was conducted before and after the reindexing, using 40 search questions from an initial user survey and two information professionals as judges for the formulation of search queries and relevance judgment of retrieved documents. This study was a case study of the potential search effectiveness of thesaurus terms. The results show that the average precision for initial queries across all analyzed search queries after reindexing increased from 0.38 to 0.50, whereas the average recall for initial queries also increased from 0.25 to 0.37. The differences in precision and recall were statistically significant. This finding suggests that the process of reindexing, supplementing the bibliographic records with thesaurus terms, substantially improved the search effectiveness of a research repository in terms of both precision and recall measures. It should be noted, however, that two experienced information professionals with subject indexing expertise formulated the search queries. As such, ordinary users might not be able to achieve the same level of performance. And IR system differences may also affect search performance, which will be further discussed by drawing from a meta-analytic study by Liu [5].

The Liu study was a meta-analysis of the effect of search experience on user search performance in terms of the recall measure. The method of meta-analysis, widely employed in the biomedical domain, was used to aggregate research findings quantitatively to reach general conclusions. It has been believed that experienced searchers will be able to draw upon their search skills to obtain better search outcomes. This meta-analytic study was designed to estimate the effect of search experience on search performance in controlled IR user experiments. The results from eight empirical user experiments, based on specific selection criteria, indicate that search experience overall has an overall positive but small effect on the recall measure (weighted mean correlation coefficient r = 0.04). The hypothesis that experienced searchers will perform better than novice searchers in terms of the recall measure was not supported. Overall, these results have suggested that search experience as one of the extensively studied user characteristics has not significantly affected search outcomes.

Drawing from a study that was designed to assess the effectiveness of MeSH terms when used by different types of searchers in an interactive search environment, Liu [6] also assessed the impact of document representations with or without MeSH terms on user search performance. This study assessed potential search effectiveness by comparing the search results from an experimental IR system, with or without an additional layer of document representation, using the original queries that had not been formulated to include MeSH terms. The results suggest that adding MeSH terms to search indexes in a Boolean-based, ranked IR system generally will not produce better search outcomes in terms of the precision and recall measures.

The results from Hider et al and Liu [4], [5] and [6] generally have indicated that the nature of the controlled vocabulary, the IR system and the skill of indexers and searchers affect the search outcomes. For different IR evaluation purposes, the identified factors should be considered through rigorous research design and procedures, particularly when human participants are involved in an IR evaluation study. The role of user characteristics will be discussed in more details below.

Finally, in an eye-tracking study of user interactions with IR systems, Liu, Thomas, Gedeon, Bacic & Li [7] investigated domain experts’ interactions with search interfaces within the context of biomedical information searching, with a goal of improving search user interface design. Specifically, this study examined the relationship between user characteristics (domain knowledge, search experience and cognitive style) and gaze behavior, as well as user perceptions of search interfaces. A total of 32 users participated and searched for documents answering eight complex questions, using four different search interfaces. The findings suggest that types of search interfaces and user-perceived search task difficulty have significant effects on gaze behavior in terms of fixation-based measures of areas of interest (AOI), that is, visual attention to the elements of title, author, abstract and MeSH (Medical Subject Headings) terms in document surrogates. MeSH terms received more attention when displayed alongside each document for experienced searchers.

Further analysis by Wittek, Liu, Daranyi, Gedeon & Lim [8] has revealed that

  • when users perceived their search tasks as difficult, they did not attend to all content elements in documents;
  • searchers with different cognitive styles may use different search strategies under an environment with uncertainty they perceive as difficult; and
  • certain search behavior types, such as issuing queries and MeSH terms that involve notable mental efforts and exploitation of resources, are correlated with changes in eye gaze patterns.

So user characteristics of search experience and cognitive style affect gaze and search behavior, when the search tasks are perceived a difficult and there are limited mental resources.

Summary

Drawing from several IR user experiments, this article has briefly discussed the usefulness of controlled vocabulary from IR perspectives and, more importantly, demonstrated how the controlled vocabulary, IR system and the skill of indexers and searchers affect search outcomes. Recent eye-tracking study results have further revealed that search experience and cognitive style, as well as perceived search task difficulty, affect how users interact with search system user interfaces. The usefulness of controlled vocabulary is still an unanswered research question.

Resources Mentioned in the Article

[1] Vickery, B. C. (1970). Document description and representation. Annual Review of Information Science and Technology, 6, 113-140.

[2] Calhoun, K. (2006). The changing nature of the catalog and its integration with other discovery tools. Retrieved from www.loc.gov/catdir/calhoun-report-final.pdf

[3] Svenonius, E. (1986). Unanswered questions in the design of controlled vocabularies. Journal of the American Society for Information Science, 37(5), 331-340.

[4] Hider, P., Dalgarno, B., Bennett, S., Liu, Y.-H., Gerts, C., Daws, C., . . . Macaulay, R. (2016). Reindexing a research repository from the ground up: Adding and evaluating quality metadata. Australian Academic & Research Libraries, 47(2), 61-75. doi:10.1080/00048623.2016.1204589

[5] Liu, Y.-H. (2010). A meta-analysis of the effects of search experience on search performance in terms of the recall measure in controlled IR user experiments. In F. Scholer, A. Trotman, & A. Turpin (Eds.), Proceedings of the Fifteenth Australasian Document Computing Symposium (pp. 105-110). Melbourne, Australia: School of Computer Science and IT, RMIT University.

[6] Liu, Y.-H. (2010). On the potential search effectiveness of MeSH (Medical Subject Headings) terms. In Proceedings of the Third Symposium on Information Interaction in Context (IIiX ’10) (pp. 225-234). New York: ACM.

[7] Liu, Y.-H., Thomas, P., Gedeon, T., Bacic, M., & Li, X. (2016). Natural search user interfaces for complex biomedical search: 2014 ALIA Research Grant Award final report.

[8] Wittek, P., Liu, Y.-H., Daranyi, S., Gedeon, T., & Lim, I. S. (In Press). Risk and ambiguity in information seeking: Eye gaze patterns reveal contextual behavior in dealing with uncertainty. Frontiers in Psychology.


Ying-Hsang Liu is a lecturer in information management at School of Information Studies, Charles Sturt University (CSU), and a visiting fellow at the College of Engineering & Computer Science, The Australian National University. He can be reached at yingliu@csu.edu.au