Quick Website Diagnostics with Search Analytics

I have recently been giving courses directed to web editors on how to successfully apply search technology on a public web site. One of the things we stress is how to use search analytics as a source of user feedback. Search analytics is like performing a medical checkup. Just as physicians inspect patients in search of maladious symptoms, we want to be able to inspect a website in search of problems hampering user experience. When such symptoms are discovered a reasonable resolution is prescribed.

Search analytics is a vast field but as usual a few tips and tricks will take you a long way. I will describe three basic analysis steps to get you started. Search usage on public websites can be collected and inspected using an array of analytics toolkits, for example Google Analytics.

How many users are using search?

For starters, have a look at how many of your users are actually using search. Obviously having a large portion of users doing so means that search is becoming very important to your business. A simple conclusion stemming from such evidence is that search simply has to work satisfactorily, otherwise a large portion of your users are getting disappointed.

Having many searchers also raises some questions. Are users using search because they want to or because they are forced to, because of tricky site navigation for example? If you feel that the latter seems reasonable you may find that as you improve site navigation your number of searchers will decrease while overall traffic hopefully increases.

Just as with high numbers, low numbers can be ambiguous. Low scores especially coupled with a good amount of overall site traffic may mean that users don’t need search in order to find what they are looking for. On the other hand it may mean that users haven’t found the search box yet, or that the search tool is simply too complicated for the average user.

Aside from the business, knowing how popular search is can be beneficial to you personally. It’s a great feeling to know that you are responsible for one of the most used subsystems of your site. Rub it in the face of your colleague!

From where are searches being initiated?

One of the first recommendations you will get when implementing a search engine for your web site is to include the search box on each and every page, preferably in a standardized easy-to-find place like the top right corner. The point of having the search box available wherever your users happen to be is to enable them to search, typically after they have failed to find what they are looking for through browsing.

Now that we know that search is being conducted everywhere, we should be keeping an eye out for pages that frequently emit searches. Knowing what those pages are will let us improve the user experience by altering or completing the information there.

Which are the most common queries?

The most frequently issued queries to a search system make up a significant amount of the total number of served queries. These are known as head queries. By improving the quality of search for head queries you can offer a better search experience to a large amount of users.

A simple but effective way of working with search tuning is this. For each of the 10, 20 or 50 most frequent queries to the system:

  1. Imagine what the user was looking for when typing that query
  2. Perform that query yourself
  3. Examine the 5-10 top results in the result list:
    • Do you think that the user was content with those results
    • If yes, pat your back 🙂
    • If not, tweak using synonyms or best bets.

Go through this at least once a month. If the information on your site is static you might not need to change a lot of things every time, but if your content is changing or the behavior of the users you may need to adjust a few things.

Systematic Relevance: Evaluation

Perfect relevance is the holy grail of Search. If possible we would like to give every user the document or piece of information they are looking for. Unfortunately, our chances of doing so are slim. Not even Google, the great librarian of our age, manages to do so. Google is good but not perfect.

Nevertheless, as IT professionals, search experts and information architects we try. We construct complicated document processing pipelines in order to tidy up our data and to extract new metadata. We experiment endlessly with stop words, synonym expansion, best bets and different ways to weigh sources and fields. Are we getting any closer? Well, probably. But how can we know?

There are a myriad of knobs and dials for tuning in an enterprise search engine. This fact alone should convince us that we need a systematic approach to dealing with relevance; with so many parameters to work with the risk of breaking relevance seems at least as great as the chance of improving on it. Another reason is that relevance doesn’t age gracefully, and even if we do manage to find a configuration that we feel is decent it will probably need to be reworked in a few months time. At Lucene Eurocon Grant Ingersoll also said that:

“I urge you to be empirical when working with relevance”

I favor the trial and error approach to most things in life, relevance tuning included. Borrowing concepts from information retrieval, one usually starts off by creating a gold standard. A gold standard is a depiction of the world as it should be: a list of queries, preferably popular or otherwise important, and the documents that should be present in the result list for each of those queries. If the search engine were capable of perfect relevance then the results would be 100% accuracy when compared to the gold standard.

The process of creating such a gold standard is an art in itself. I suggest choosing 50 or so queries. You may already have an idea of which ones are interesting to your system; otherwise search analytics can provide this information for you. Furthermore, you need to decide which documents should be shown for each of the queries. Since users are usually only content if their document is among the top 3 or 5 hits in the result list, you should have up to this amount of documents for each query in your gold standard. You can select these documents yourself if you like. However, arguably the best way is to sit down with a focus group selected from among your target audience and have them decide which documents to include. Ideally you want a gold standard that is representative for the queries that your users are issuing. Any improvements achieved through tuning should boost the overall relevance of the search engine and not just for the queries we picked out.

The next step is to determine a baseline. The baseline is our starting point, that is, how well the search engine compares out of the box to the gold standard. In most cases this will be significantly below 100%. As we proceed to tune the search engine its accuracy, as compared to the gold standard, should move from the baseline toward 100%. Should we end up with accuracy below that of the baseline then our work has probably had little effect. Either relevance was as good as it gets using the default settings of the search engine, or, more likely, we haven’t been turning the right knobs.

Using a systematic approach like the one above greatly simplifies the process of working with relevance. It allows us to determine which tweaks are helpful and keeps us on track toward our ultimate goal: perfect relevance. A goal that, although unattainable, is well worth striving toward.

Knowledge Management: Retrieve, Visualize and Communicate!

As noted by Swedish daily paper Metro, Findwise is working with JCDEC, the Joint Concept Development & Experimentation Centre at Swedish Military Headquarters. In Metro’s words the project aims at developing a knowledge management system for the headquarters of tomorrow. The system is expected to be up and running in time for the international military exercise VIKING 11, to be executed in April of 2011.

Good decisions stem from good information; this is true for both military and civilian enterprises. Vast amounts of time and resources are being invested in order to collect information. But to what end? Granted, somewhere among that information there is probably something you will find useful. But large amounts of information quickly become incomprehensible. In order to combat information overload you need a select-and-filter tool such as Search, and that’s where Findwise comes in.

However, for JCDEC it is not enough to simply locate the information they have available. Captain Alexandra Larsson, Concept Development Lead for Knowledge Support, makes this fact very clear. It is just as important to get an idea of what information is not there. In essence, JCDEC is in the process of creating information from information. This is also one of the great differences between the kind of web-based search and retrieval systems we have come to depend on and a state of the art knowledge management system. The latter is not just a retrieval tool; it is an information workbench where the user can select, retrieve, examine and manipulate information.

The key to finding information gaps is to study patterns. For example, consider the trivial problem of birthday distributions. Without any prior knowledge one would probably expect there to be roughly as many births in May as in August or November. This is not always the case. Depending on where you are in the world birth figures may actually be skewed so that one month has significantly more births than other months do. Why does this happen? Being able to pose that exact question may in turn teach us a lot about the mysterious workings of the Stork.

In military intelligence the filling of information gaps may mean the difference between victory and defeat. Why is there an increase in partisan activity in that district? Why were eight weapons silos raided over the course of two days? Why at this moment in time? These questions are expected to lead to insights into the plans and activities of suspects and to notify those in command of looming threats.

Retrieve, Visualize, Communicate

Retrieval

The envisioned work-flow for JCDEC information operators is threefold: retrieval, visualization and communication. Each research session will typically be initiated through a keyword search interface, much like you would issue a web query on Google. Just like its online counterpart the system would present the results ordered according to their expected relevancy to the operator’s query. Using facets and query refinement the result set can be narrowed down until the information in front of the operator is expected to contain that which is being sought for.

Captain Alexandra Larsson hints at another strategy for getting to information. Facets are so speedy these days that they can be applied on the full document set without any delays. Clearly, JCDEC is using search technology to provide directory listings much like websites such as the Open Directory Project, although completely dynamic. The option of simply browsing these directories is also available to operators.

Visualization

The next step, visualization, employs an array of tools for visually displaying the results. These include plotting objects on maps and timelines and looking for groupings where objects have a disproportionately dense distribution, so called cluster analysis, among others. This is where clues are uncovered and questions posed: why there, at that time, with those people? In some cases a field investigation is necessary in order to answer these questions. Other times the answers can be deduced from the tools themselves. The tools also allow the operator to formulate new search queries based on the visual information. The operator may choose to limit the scope of the search to one or more of the clusters in the timeline or map, for example.

Communication

If or when the operator finds something interesting this should be recorded. But to JCDEC it is not necessarily the results themselves that are important. The act of getting to the information is valuable in itself. The reason for this is that different operators have different backgrounds and possess different types of information. Where one operator filters or deduces information from a search result in one way, another operator might choose a completely different approach and unveil other clues.

According to Captain Alexandra Larsson it is absolutely necessary that operators share knowledge as well as refinement strategies as part of their work. One of the paradigms that JCDEC is looking to experiment with is social bookmarking along with the ability to search through sets of bookmarked objects. Objects can be both tagged and commented on, useful for conveying meta-information to fellow operators. It is likely that there will be custom-based filters, where an operator can inform the system of types of objects that do and do not interest him or her and have the system automatically filter the result sets based on this information. These filters can of course also be shared with other operators.

An evolving system

The process of retrieval, visualization and communication is only one, albeit the most prominent, feature of the JCDEC knowledge management system. The system itself will be put to use in the spring of 2011 and development will surely continue beyond that point. The ideas and concepts at work today will most likely be refined over time as Captain Alexandra Larsson and JCDEC learn from hands-on experience with working with information. And as evolution progresses I hope to be able to go into more detail on some of the other tidbits.