Abstract
What elicitations or requests for information do human search intermediaries make to patrons search with information requests that require an online search? What intermediary elicitations occur before and during an online search? Why do human search intermediaries make elicitations? This study reported in this paper investigated search intermediary elicitations during mediated online searching. A study of 40 online reference interviews involving 1557 search intermediary elicitations, found 15 different types of search intermediary elicitations to patrons. The elicitation purposes included search terms and strategies, database selection, relevance of retrieved items, and patrons' knowledge and previous information-seeking. Analysis of the patterns in the types and sequencing of elicitations showed significant strings of multiple elicitations regarding search terms and strategies, and relevance judgments. This paper discusses the implications of the findings for training search intermediaries and the design of interfaces eliciting information from end-users.
Introduction
Question asking or elicitation behavior, as requests for information, form a significant part of day to day communication between individuals. Elicitations by search intermediaries form part of the dialogue through which search intermediaries develop a model of the user's goals, intentions and background (Belkin, 1984). Online searches conducted by professional search intermediaries may take place with or without the user present during the online interaction. This paper reports findings from a study of search intermediary elicitations of patrons present during the pre-online reference interview and the online search itself.
Patron Elicitations
Before a recent study by Wu (1992) limited research existed concerning patrons elicitation behavior during the process of mediated online searching. Using discourse, content and statistical analysis from 38 patron /search intermediary interactions, Wu identified 861 patron elicitations. She also identified 10 categories of patron elicitation purposes, including search terms, databases, search procedures, current/immediate actions, outputs and social topics. Wu found that patrons made more elicitations to intermediaries, than intermediaries made of patrons during the online search stage. Various situational and individual factors were found to affect patrons' elicitation behavior.
Search Intermediary Elicitations
There have been many empirical and theoretical investigations of the patron - librarian interaction during the reference interview. Many studies have focused on the benefits of librarians asking open questions with little structure as opposed to closed questions with a tight structure during reference interviews (Katz, 1987; King, 1972; Lynch, 1978; McFayden, 1975; Taylor, 1968). Researchers have examined many aspects of online reference interviews, including neutral questioning (Dervin & Dewdney, 1986), evaluation of intermediaries' communication behavior (Auster & Lawton, 1984; Markey, 1981), and dialogue between search intermediaries and patrons (Cochrane, 1981; Crouch & Lucia, 1981; Horne, 1990; Ingwersen, 1982).
Brooks (1986) investigated the features of an intelligent interface simulating the functional behavior of a good search intermediary. Saracevic, Mokros and Su (1990) examined the major cognitive activities during patrons, search intermediaries and IR systems interaction. Research studies have also explored the major functions required for information provision by information retrieval systems (Belkin, Seeger & Wersig, 1983; Belkin, Brooks & Daniels, 1987; Vickery, 1985).
Kuhlthau, Spink & Cool (1992) studied search intermediary elicitations regarding the patron's information-seeking. They found that search intermediaries made few elicitations regarding patrons' previous information-seeking behavior related to their information problem. These studies have expanded our understanding of the role of search intermediaries, although previous research has not fully investigated the elicitation purposes of search intermediaries. The current study builds on the previous studies by Kuhlthau, Spink and Cool and Wu, by examining intermediary elicitation behavior before and during mediated online searches.
Research Questions
Specific research questions addressed in this study were:
1) What were the purposes of search intermediary elicitations?
2) When during the interaction with the patron do
intermediary elicitations occur?
3) What were the sequences of intermediary elicitation occurrences?
To address these research questions, a unit of elicitation analysis, stages of the interactions and sequences of intermediary elicitation occurrences were identified.
Research Design
Data Corpus
The current analysis used data collected during a previous study by Saracevic and Su (1989), including the transcribed discourse between forty patrons by one of four intermediaries at Rutgers University Libraries. The search topics included humanities, social sciences, sciences, and medicine, using real online searches, with no restrictions on length or topic. Experienced professional searchers performed online searches within the area of their subject expertise. The pre-online interviews and online searches were videotaped. As the purpose of the current study was to identify and classify intermediary elicitations to the patrons, the transaction logs proved sufficient for the analysis. A summary of the data corpus appears in Table 1.
Table 1. Summary of data corpus characteristics
Searches: Number of searches 40 Hours of videotape 46.05 hrs. Average time per search 69.08 min. Average time of pre-online phase 13.04 min. Average time of online phase 56.04 min. Search Intermediaries: Number of search intermediaries 4 Average experience of search intermediaries 8.5 years Item retrieval: Number of items retrieved 6225 Number of relevant (or partially relevant) items retrieved 3565 Number of non-relevant items retrieved 2660 Average relevant (or partially relevant) items per search 99 Average non-relevant items per search 67 Databases searched: Number of databases searched 46 Average number of databases consulted per search 3
Definition of Terms
This study focuses on the questions, i.e., elicitations, made of patrons by search intermediaries. The primary unit of analysis was an elicitation. Utterances are the secondary unit of analysis, but were the first unit encountered by the researchers. An utterance is any uninterrupted verbalization by a participant in the dyadic user-intermediary dialogue. If a verbalization by party A was interrupted by a verbalization produced by party B, after which party A finished the original verbalization. Then three distinct utterances had occurred in this fashion: A: verbalization... followed by B: verbalized interruption and then A: ...continued verbalization.
The second unit of analysis in the transaction logs was an elicitation, defined by Wu (1992) as "a verbal request for information" (p. 32). During the dyadic conversations, an elicitation was an utterance by person A (i.e., intermediary), which produced (i.e., elicits) a response from person B (in this case, a patron), and for which, it was the intent of the intermediary to do so. To judge the intermediary's intent, strong evidence of intent had to be present in the discourse transcripts. An utterance had to generate a response from a patron to be an elicitation.
Rhetorical questions for which no response was expected from the patron were not elicitations (e.g., "You know what I'd like to do?"). If the search intermediary did not clearly intend (based on evidence in the transaction log) to elicit a response from the patron, the utterance(s) were not counted as an elicitation.
It was possible that several utterances by a search intermediary formed one elicitation. For example, utterances 1 and 3 above constituted a single intermediary elicitation to a patron. This was interrupted by the patron. Then utterances 1 and 3 were counted as a single elicitation.
Methodology
In a fashion similar to Saracevic, Mokros, and Su (1990), this study employs a grounded theory approach to the analysis of the data (Strauss & Corbin, 1990). Using the same data as discussed in the Data Corpus section above, Wu (19) found the following 10 categories of patron elicitations. These categories were also used as an initial foundation for the present study: (1) terms, (2) databases, (3) search procedures, (4) other online issues, (5) other information issues, (6) current/immediate action or plan, (7) outputs, (8) echoic, (9) social and meta-topics, and (10) indiscernible or unfinished.
As the task of intermediaries differs from that of the patron, it was expected that the intermediary elicitation purposes would both differ and parallel patron elicitation purposes. For that reason, the problem of the intermediary elicitation categories was approached from a descriptive framework and an initial starting point of Wu's (1992) categories. The analysis of intermediary elicitation purposes was an iterative process conducted in four phases:
1. Identify search intermediary elicitations in the 40 cases, and where possible, fit them to Wu's categories
The three researchers (coders) coded one search together to get a feel for the data and how each coder would approach the coding problem. The remaining 39 searches were divided into three groups of 13 searches. Each researcher then coded 13 searches. Our first step was to determine the number of intermediary elicitations in each search. To identify an elicitation a coder needed explicit evidence from the transcript that the intermediary was seeking information in the form of a patron response. Utterances were tagged as elicitations by a coder where appropriate. Wu's (1992) categories were then consulted to allocate an existing categorize to the elicitation. If so, the coder could move to the next utterance. If not, the coder made notes in the margin suggesting an appropriate category for the elicitation.
2. Rework the categories to make them appropriate to the search intermediary point of view
The three researchers then met to discuss the coding of the data. Suggestions for the revision of categories to reflect search intermediary processes were discussed until a consensus of the revised categories was reached.
3. Re-analyze the 40 cases based on the new categories of search intermediary elicitations
The search intermediary elicitations were then re-analyzed based on the categories listed above and re-categorized under the new scheme. Since the criteria for an utterance to be an elicitation had not changed, this re-analysis was a matter of re-coding existing elicitations.
4. Check for inter-coder agreement
Inter-coder agreement is the similarity between each coder regarding, (a) whether an utterance was an elicitation, and (b) which category an elicitation should be placed. To check coding consistency, each coder chose 4 searches from his or her group of 13 searches to give another coder. Each researcher coded an additional eight searches. Disagreements regarding appropriate codes were settled at a final meeting of all coders.
Loglinear Analysis
After categorizing the search intermediary elicitations in the 40 online searches, we then analyzed the sequence in which the elicitations occurred in each case. Loglinear analysis (Mokros, 1985; Knoke & Burke, 1979) was used to determine the sequential occurrence of the search intermediary elicitations to determine which transitions from one category of elicitation to another, if any, occurred with significant frequency (i.e., non-random transitions).
Results
One thousand five hundred and fifty seven intermediary elicitations were identified and were further divided into 15 different elicitation purposes occurring at two stages of the search.
Table 2 details the number and percentage of search intermediary elicitations with each elicitation purpose category - pre-online and online stage of the search.
Table 2. Number and percentage of search intermediary elicitations within each elicitation purpose category - presearch stage and online stage (Number of searches = 40 ).
Categories of Search Intermediary Elicitation Purposes
Each category of search intermediary elicitation purpose is described below with an accompanying example.
User Domain Knowledge (UDK): A request information
regarding the patron's knowledge of the domain of the topic at
hand. Example from
Search No. 7: "...when there is an author writing on
this topic will they use cyanogenic glycoside or will they
mention something
else?"
Databases (D): Databases to be used, including database
subject coverage, and restrictions. Example from Search No. 21:
Would you like to
"try Psych Abstracts first or would you like to try the
educational first?"
User Knowledge of Database Search (UDS): Seeks to
determine the patrons' knowledge of database searches (whether
online or CD-ROM).
Example from Search No. 10: "Have you done online searching
before?"
User Information Seeking (UIS): The patrons' information
seeking stage on this topic. Example from Search No. 27:
"...clearly you know a
lot of the literature, thus far you haven't found
anything?"
Search Strategy and Terms (SST): An elicitation relating
to the concepts or terminology of the patron's query in order
to formulate or
reformulate the search strategy or generate key words
whether occurring at the pre-search stage or during the search
stage. Includes
languages to be searched and time period to be covered by
the search. Example from Search No. 7: "Why don't you tell me
what you meant by
insect adaptation to host plants."
Other Online Issues (OOI): General aspects of online
database searching such as online fees and
telecommunications. Example from
Search No.6: "Well, I'm just reading the last sentence of
what you wrote down. It would cost extra.."
Output Magnitude (OM): Relates to the number of items
retrieved by the system in response to the search query.
Example from Search No. 36:
"We didn't get rid of a lot, did we?"
Output Relevance (OR): Relates to the evaluation of items
retrieved by the system. Example from Search No. 20 "But the
references we've
come up with so far, do you think will be useful?"
Output Format (OF): Relates to the format of outputs,
i.e.: citations, abstracts, or full-text. Example from Search
No. 2: "...the
question is if you want citation or full records..."
Output Terms (OT): Refers to terms generated by the
system during the search phase including descriptors. Example
from Search No.
6: "I found the term it's extras-systol. Is that the same
thing or.."
Technical Aspects of Search (TECH): Technical aspects of the
search. Includes elicitations about logging on, printer set-up,
and computer
commands. Example from Search No. 2: "Do you have a disk for
me?"
Other Information Services (OIS): Other sources of
information for the patron. (e.g., interlibrary loan,
microfiche, colleagues). Example
from Search No. 25: "Do you know about interlibrary loan
here?"
Echoic (E): Requesting the repetition or reinforcement of a
previous utterance. Example from Search No. 27: "Seven,
yes?"
Social/Metatopic (SQ): Non-search related social
elicitations. Example from Search No. 18: "You know what I
really want to know is, how's
your sister?"
Indiscernible (I): An elicitation that is either muted,
unfinished, or interrupted to such an extant that it is
indiscernible for this
analysis. Example from Search no. 20: "Does it sound like
(inaudible)"
Table 2 displays a total of 645 presearch elicitations (41.4 %) and 912 online elicitations (58.6%) with a mean number of presearch elicitations per search of 16, a mean number of online elicitations per search of 23, and a mean number of total elicitations per search of 39. Wu's (1992) analysis of the data differed somewhat, as she found a total of 331 presearch intermediary elicitations (mean 9) and 530 online intermediary elicitations (mean 14) with a total intermediary elicitation count of 1713 (mean 45).
Table 2 further illustrates the following:
1. The percentage of search intermediary elicitations at the online search stage was larger than inthe pre-search stage (41. 4 % presearch, 58.6% online stage).
2. The largest number of search intermediary elicitations were related to search strategy and terms, followed by the online relevance elicitations (combined total of approx. 66 % of all elicitations).
3. A total of 383 (25 %) search intermediary elicitations occurred in one of the four online output purpose categories: i.e., output relevance, output magnitude, output terms, output format.
4. Search strategy and terms accounted for the greatest percentage of either presearch or online elicitations (50 % total) - of the 645 total presearch elicitations, 453 (73 %) related to search strategy and terms.
Table 3 shows the number of online searches which included each category of search intermediary elicitation.
None of the 40 online searches contained all 15 categories of elicitation purposes and the mean number of categories per search was 7 within a range of 4-11. All online searches included the search strategy and terms elicitations and 95 % of the searches included online relevance elicitations.
Sequences of Elicitations
Analysis was also conducted on the patterns in the types and sequencing of search intermediary elicitations showing statistically significant transitions between search strategy and terms elicitations and online relevance elicitations. The analysis indicates a high frequency of strings (or a series) of search strategy and terms elicitations and strings of online relevance elicitations to patrons. Online relevance elicitations were most frequently followed by search strategy and term elicitations.
Discussion
These preliminary results from the analysis of the search intermediary elicitations begins to provide a better understanding of the search intermediaries' goals and purposes during mediated online searching, and suggest what search intermediaries (whether human or expert system) need to know throughout the mediated online searching process.
Wu (1992) found that search intermediaries performed patron modeling functions mainly during the pre-search stage and made more elicitations during this stage than the online stage, implying that search intermediaries pre-planned their elicitations. The current analysis shows that search intermediaries made more elicitations during the online stage than the pre-online stage of the search, indicating that search intermediaries continued to dynamically model the patron's information problem throughout their conversation with the patron and maintain a constant focus on search terms and strategy. This supports the terminological determinant nature of the mediated online search process identified by Saracevic, Mokros, Su & Spink (1991). The major focus of the search intermediary was to elicit search terms from the patron, not only during the pre-online strategy formulation, but also during the online search strategy reformulation stage.
The second major focus of the search intermediaries, particularly during the online search stage, was the elicitation of relevance judgements from the patron on the retrieved items. This supports Spink (forthcoming), who found search intermediary prompted patron relevance feedback loops and judgements to be a major determinant during mediated online searching.
Other purposes represented a small percentage of the elicitations by search intermediaries, including elicitations regarding the patron's domain knowledge or previous information-seeking related to their information problem. This support Kuhlthau, Spink & Cool (1992) who found that search intermediaries asked few direct questions of patrons related to their information seeking and work stage. They did find that patrons often made unsolicited statements regarding their information-seeking and work stage during their conversation with a search intermediary. The data from real mediated online searches does not seem to support a model of the search intermediary as attempting to develop a model of the patron's background, goals or intentions, and the state of their information problem (Belkin, 1984).
Conclusion
Results of this study show that search intermediaries engage in a complex process of elicitation behavior throughout the mediated online searching process. Further research is needed to model the interactive elicitation behavior of the search intermediary and the patron. The findings indicate that elicitations to patrons constitutes a potentially important element in the mediated search process.
Acknowledgment
The author thank Dr. Mei Mei Wu for her previous work. This study was funded by a University of North Texas Summer Research Award 1994.
References
Auster, E., and S.B. Lawton. 1984. Search interview techniques and information gain as antecedents of user satisfaction with online retrieval. Journal of the American Society for Information Science 35: 90-103.
Belkin, N.J. 1980. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science 5:133- 143.
Belkin, N.J. 1984. Cognitive models and information transfer. Social Science Information Studies 4:111-129.
Belkin, N. J., H.M. Brooks, and P.J. Daniels. (1987). Knowledge elicitation using discourse analysis. International Journal of Man-Machine Studies 27:127-144.
Belkin, N. J., T. Seeger, and Wersig, G. 1983. Distributed expert problem treatment as a model for information system analysis and design. Journal of Information Science 5:153-167.
Brooks, H. M. 1986. An intelligent interface for document retrieval systems: Developing the problem description and retrieval strategy components. Unpublished Ph.D. dissertation. Department of Information Studies, The City University, London.
Cochrane, P. 1981. Study of events and tasks in pre-search interviews before searching. Proceedings of the 2nd National Online Meeting, 133-147.
Crouch, W., and J. Lucia. 1981. Analysis of verbal behaviors in the pre-search interview. Report of the Presearch Interview Project. Syracuse University, N.Y., School of Information Studies (ED 205 184).
Dervin, B., and P. Dewdney. 1986. Neutral questioning: A new approach to the reference interview. RQ 25:506-512.
Horne, E. 1990. An investigation into self-questioning behavior during problem-solving. Proceedings of the 53rd Meeting of the American Society for Information Science, 27: 86-90.
Ingwersen, P. 1982. Search procedures in the library: Analyzed from the cognitive point of view. Journal of Documentation 38(3):165- 191.
Katz, W. A. 1987. Introduction to reference work, Vol.2: Reference services and reference processes. New York: McGraw-Hill.
King, G. 1972. The reference interview: Open and closed questions.RQ 12:157-160.
Kuhlthau, C. C. 1991. Inside the search process: Information-seeking from the user's perspective. Journal of the American Society for Information Science 42(5):361-371.
Kuhlthau, C. C., A. Spink, and C. Cool. (1992). Exploration into stages in the information search process in online information retrieval: Communication between patrons and intermediaries. Proceedings of the 55th Annual Meeting of the American Society for Information Science 29:67-71.
Lynch,M.J. 1978. The reference interview in public libraries.Library Quarterly 48:119-142.
Markey, K. 1981. Levels of question formulation in negotiation of information need during the online presearch interview: A proposed model. Information Processing and Management 17(5):215-225.
Mokros, H. B. 1985. Patterns of persistence and change in action sequences. In S. Duncan, D.W. Fiske, R. Denny, B. Kanki, and H. B. Mokros. (Eds.), Interaction structure and strategy. (pp. 175-232). Cambridge, MA:Cambridge University Press.
Saracevic, T., H.B. Mokros and L. Su. 1990. Nature of interaction between users and intermediaries in online searching: A qualitative analysis. Proceedings of the 53rd Annual Meeting of the American Society for Information Science 27:47-54.
Saracevic, T., H.B. Mokros, L. Su, L, and A. Spink. 1991. Interaction between users and intermediaries in online searching. Proceedings of the 12th National Online Meeting, 329-342.
Saracevic, T., and L. Su. 1989. Modeling and measuring user-intermediary-computer interaction in online searching: Design of a study. Proceedings of the 52nd Annual Meeting of the American Society for InformationScience 26:75-80.
Spink, A. (forthcoming).A user-centered feedback model for information retrieval.
Strauss, A., and J. Corbin. 1990. Basics of qualitative research: Grounded theory procedure and techniques. Newbury Park: Sage Publications.
Taylor, R. S. 1962. The process of asking questions. American Documentation 13:391-396.
Taylor, R. S. 1968. Question-negotiation and information-seeking in libraries.Report 3: Studies in the Man-Machine Interface in Libraries Bethleham, PA:Lehigh University.
Vickery, A. 1985. Question analysis. In N. J. Belkin and A. Vickery (Eds.), Interaction in information systems. England: Cambridge University (pp.20-43).
Wu, M. 1992.Information interaction dialog: A study of patron elicitation in the information retrieval interaction.Unpublished doctoral dissertation. School of Communication, Information and Library Studies, Rutgers University