The Project

Examining Query Logs

Introduction

Do people do what they say they do? In our interviews with a whole variety of folks for the MaNIS Interface Design project - mammalogists, wildlife biologists, students - we arrived at some intruiging conclusions that shaped features of our prototype's interface design.

For example, despite our attempts to identify people interested in begining their searches with a geographic location, all of our interviewees claimed they'd start by specifying a taxon - hence the default to the Taxon panel in our prototype. Similarly, everyone was interested in searching all of the collections in the network, with the exception of curators looking up something in their own collections, or researchers downloading data by museum location. Users did not describe any search tasks where they would want to specify some subset of museums from the network that was larger than one but fewer than all. So we defaulted our prototype design to search all museums on the network, and emphasized one-click selection of individual museums on the prototype's Institution panel.

But we only talked to about 10 people...will these patterns hold across a larger user base? Also, the MaNIS Interface Design project interviews focused on mostly non-curators using mammal collections. Through query log studies and interviews with curators and people using herp, bird, insect, plant and fossil collections, I hope to identify any important differences and the implications these have on the design of the online browse/search interface.

The Query Log Studies attempt to quantify user behavior and determine if it supports or conflicts with the results of the interviews. They also quantify the frequency of use of various features of the interfaces, and attempt to characterize specific search/browse patterns. These patterns become typical task flows that the ideal search/browse interface should support readily. Finally, the study identifies various types of requirements for new Query Log applications.

Content

  • Guiding Questions
  • Methods and Data
  • Results
  • Specifications for Ideal Query Logs

Guiding Questions

During the study, I attempted to use the data in the query logs to shed light on the following questions.

Characterizing Search Behavior

  • How often are geographic search terms used? Taxonomic? In combination?
  • How often are non-geographic, non-taxonomic search conditions used?
  • What characterizes queries which pull up no results? (zero-match queries)
  • How many different queries does the 'average' user submit in a searching session?
  • In what order during the searching session are the fields added to the query specification?
  • How many search conditions are typically used?

Examining Interface-Specific Effects

  • How often is field x used for searching?
  • With what frequency are interface features used or defaults changed?
  • MaNIS: how many institutions do people select to search?
  • What differences are seen with different interface designs?

Identifying Behaviors of User Groups

  • Can particular types of multi-query searching sessions be characterized? Can this be usefully visualized?
  • Do museum curators use specimen databases differently from other users? What are these differences?
  • Do people working with different types of collections show different behaviors?
  • Can we tell anything about what the public wants from a search interface?

Identifying Requirements for Query Log Viewing Applications

  • Of the query log applications examined, which features supported the study? Which features hindered it?
  • Which questions were answerable from which applications? Which weren't?
  • What other interesting data or questions emerge?
  • What aspects provide feedback about the users, what about the collections' use, what about the interface?
  • What recommendations do I have for future query log viewing applications?

integrate these Query Questions <unfinished> :

  • To what extent do people use fields OTHER than scientific name or geographic? (MQ: to what extent is this influenced by not seeing that info on the search results page??)
  • What's happening when people don't get any matches? Is it spelling? conflicts in entering search terms? problems with taxonomic names? Or truly nothing in the collections? Are these people correcting & trying again? Or just playing around? (easiest to tell if it's NOT "amphibia" for example)
    1. eg: dec 01, 03: putting in "opossum" but having already defaulted to amphibian, gets nothing...tries 6 searches, never getting any matches.
    2. eg: feb 01, 04: "Europe" & "israel" - there are results for either separately, (1505, 30) but not the combination - because it is non-sensical.
    3. eg: feb 01, 04: family = PANTHERA - doesn't return the results that it would in "term"; "MOOSE" returns no results b/c common name.
  • Out of x queries, how many distinct users? mean & range of queries per user? How many do They type in, vs. clicking to create?
  • what behavior scales most with the changing number of queries? That is, is it MVZ folks? Detail checking? One person or a whole bunch of people responsible for the high numbers on a given day?
  • What of this info would help a museum director/curator know how their info was helping people?
  • How many people do repeat queries to cover adjacent geographic areas or taxono names, etc? That is, how many people would use the OR feature if it were implemented?
  • how often per month or year do people use the database?(After the novelty of a new system has worn off..)
  • can we distinguish public use from academic institution use?

Methods and Data

Query logs available for study included those for the MVZ Data Access application; the raw web logs generated by the MaNIS search interface during its development; and the BNHM search/browse logs.

Daily web logs for the MVZ Data Access application were downloaded for the period from January 2003 through May 31, 2004. They consisted of a total of 140,000 queries. To achieve workable filesizes, the query records were divided by the collection they targeted - Mammals, Birds, Amphibians, Reptiles, all/Eggs/multiple.

"Making request at: " lines from the raw MaNIS program logs were collected into one file, for a total of 2377 query records from January 01 to June 10, 2004. The host information is not connected with the query records, providing some limits on the ability to recognize individual search sessions.

The "Raw data" logs of the BNHM web query log application were used.....<unfinished>

<unfinished> GBIF - usage statistics sent to Craig using MVZ's DiGir index hosted at GBIF ; BNHM , MVZ , MaNIS

Also, interviews conducted with Craig, other curators, directors about what data is needed about the use of an online access to the collections...<unfinished>

Results

MVZ: MVZ query study summary.xls

MaNIS: MaNIS query summary.xls

Specifications for Ideal Query Logs

Anticipating the development of the application for viewing the activity of the MaNIS system, and complementing the specifications for the idea search/browse interface, I provide these specifications for ideal query logs.

Museums making their collections available through the DiGIR protocol, hence without an investment in the interface design, will not be interested in the specifications that support future improvements to the online interface. The party maintaining the interface, however, may still be interested in them.

Note that there's really a fundamental difference to the logging for hyperlink-based browsing interfaces like the BNHM browse feature and Flamenco versus logging based on submission of a query form, which is the model primarily exhibited by the current MVZ and MaNIS applications. In an environment that facilitates browsing, web logs will want to identify when a user enters the application and begins a search session, then track what they add & subtract to their query over time.

Business Requirements

<unfinished>

Technology Requirements

<unfinished>

Application Requirements

<unfinished>

requirement: always have the basic data available.   bring together related information on a single line, including date, time, host, query, etc.

Make it possible to browse, but also to retrieve in large quantities if desired - possibly with a little application of code.

Think about importability into programs that will help you examine it.

MVZ logs challenges: divided by day & the date not included in the individual line.

MaNIS: timestamp good. existing tagging challenging. Need info about host.

Comments on current web-logs layout..
*would be nice to see the summary category numbers on the details page. Might also be nice to have some sort of tagging
*perhaps a useful additional statistic would be # of unique users, and average # of queries per user.
*perhaps also educational vs. non educational IPs; and the various countries...
*0 matches and 1 match are special cases. The total or average numbers of matches per query might be a useful number aside from that, but certainly shouldn't be intermingled with it...
It doesn't make sense to separate the connection between the various categories - are the MVZ folks using the detail? the egg collections? or all folks? etc.

A bonus for tracking the interface use would be a count for how often users encountered other error messages, like "can't combine eggs & tissue search".
you'd want to note if the query is one of your Example searches. You'd want to note if it was from browsing or from typing something in.
You might be particularly interested in how people change a search query.
If just one match, show it directly - currently searching on a catalog # requires two clicks, registers as two queries...

Search Interface Support Requirements

<unfinished>

Build some tracking in to help improve future iterations of the interface

  • Where are people entering the interface
  • Where leaving it
  • Which errors are they encountering
  • What do they leave with
  • Which features are used, at what rate, and how –these might become defaulted settings in future interfaces.

Information Requirements

What logs should tell directors & curators... <unfinished>

  • How many individual users?
  • Who are the users
  • Usage patterns by audience
  • Which parts of collections are used
  • common geography/taxa that people are working with?
  • What data is duplicated out in the world…
  • What collections have similar information
  • Which collections your data often is downloaded with?
  • Think about what makes sense for logs when you're not in control of the interface - what do you want to know about how your data is used?
  • How much of it is duplicated out there in downloaded files?  Which items were downloaded, so if you make corrections you know how many bad copies are out there? Or allow someone to sign up for an e-mail based alert if some of the records they download are corrected? (Or if some institutions that were offline come online? which makes more sense under the index scenario)
  • How much has the interface contributed to the improvement of the data? That is, how does it (the visualization, the tools, the hyperlink to report errors) help people who know about a taxon or area to contribute to cleaner data? How many of these contributesion have there been, how many are acted upon, etc?