Evaluate Your Search Application

Search is the worst usability problem on the web according to Peter Morville (in his book Search Patterns). With that in mind it is good to know that there are best practices and search patterns that one can follow to ensure that your search will work. Yet, just applying best practices and patterns will not always do the trick for you. Patterns are examples of good things that often work but they do not come with a guarantee that your users will understand and use search simply because you used best practice solutions.

There is no real substitute for testing your designs, whether it’s on websites intranets or any other type of application. Evaluating your design you will learn what works and does not work with your users. Search is a bit tricky when it comes to testing since there is not one single way or flow for the users to take to their goal. You need to account for multiple courses of actions. But that is also the beauty of it, you learn how very different paths users take when searching for the same information. And it does not have to be expensive to do the testing even if it is a bit tricky. There are several ways you can test your designs:

  • Test your ideas using pen and paper
  • Let a small group of users into your development or test environment to evaluate ideas under development
  • Create a computer prototype that is limited to the functionality you are evaluating
  • You can also evaluate the existing site before starting new development to identify what things need improvement
  • Your search logs are another valuable source of information regarding your users behaviors. Have a look at them as a complement.

And the best part of testing your ideas with users is, as a bonus you will learn even more stuff about your users that will be valuable to you in the future. Even if you are evaluating the smallest part of your website you will learn things that affects the experience of the overall site. So what are you waiting for? Start testing your site as well. I promise you will learn a lot from it. If you have any questions about how to best evaluate the search functionality on your site or intranet, write a comment here or drop me an email. In the meanwhile we will soon go on summer holiday. But we’ll be back again in August. Have a nice summer everyone!

The ROI of Enterprise Search—Where’s the Beef?

When faced with the large up-front investment of an Enterprise Search installation, executives are asking for proof that the investment will pay up. Whereas it is easy to quantify the value of search on an e-commerce site or as part of the company helpdesk—increased sales, shorter response times—how do you go about verifying that your million-dollar Enterprise Search application has the desired effects on your revenue stream?

Search engines on the Web have changed the landscape of information access. Today, employees are asking for similar search capabilities within the firewall as they are used to having on the Web. Search has become the preferred way of finding information quickly and accurately.

Top executives at large corporations have heard the plea and nowadays see the benefits of efficient Findability. However, it costs to turn the company information overload from a storage problem of the IT department to a valuable asset and business enabler for everybody. So how do you prove the investment worthwhile?

The Effects of Enterprise Search

Before you can prove anything, you need to establish the effects you would like your Enterprise Search solution to have on your organization. Normally, you would want an Enterprise Search solution to:

  •  Enable people to work faster
  •  Enable people to produce better quality
  •  Provide the means for information reuse
  •  Inspire your employees to innovate and invent

These are all effects that a well-designed and maintained Enterprise Search application will help you address. However, the challenge when calculating the return on investment is that you are attempting to have an effect on workflows that are not clearly visible on your revenue stream. There is no easy way to interlink saved or earned dollars to employees being more innovative.

So how do you prove that you are not wasting money?

There are two straightforward ways to address the problem: Studying how users really interact with the Enterprise Search application and asking them how they value it.

User Behavior through Search Logs

By extracting statistics from the logs of your Enterprise Search application, you can monitor how users interact with the tool. There are several statistic measures that can be interesting to look at in order to establish a positive influence on one or more of the targeted effects.

A key performance indicator for calculating if the Enterprise Search application enables people to work faster is to monitor the average ranking of a clicked hit in the result list. If people tend to scroll down the result set before clicking a hit and opening up a document, this implies the application does not provide proper ranking of the results. In other words, users are forced to review the result set, which obviously slows them down.

By monitoring the amount of users that are using the system, by following the number of different documents they open up through search and by observing the complexity of the queries they perform, you can estimate the level of information your users are expecting to find through searching.

If the application is trusted to render relevant, up-to-date results, more users will use it, they will carry out more complex queries and they will open up a wider range of different documents. If your users do not trust the system, however, they will not use it or they will only search for a limited set of simple things such as “news”, “today’s menu” or “accounting office”. If this is the case, you can hardly say your Enterprise Search application has met the requirements posed on it.

Conversely, if the users access a wide set of documents through search and you have a large number of unique users and queries, then this implies your Enterprise Search application is a valued information access tool that promotes information reuse and innovation based on existing corporate knowledge.

User Expectations through Surveys

Another way to collect information for assessing the return of investment of your Enterprise Search initiative is to ask the users what they think. If you ask a representative subset of your intended users how well the Enterprise Search application fits their specific purposes, you will have an estimate of the quality of the application.

There are a lot of other questions you can ask: Does the application help the user to find relevant corporate information? Are the results ranked properly? Does the application help the user to get an overall picture of a topic? Does it enable the user to get new ideas or find new opportunities? Does it help him avoid duplicating work already done elsewhere within the organization?

A Combination of Increased Usage and Perceived Value

As we have seen, the return on investment of an Enterprise Search initiative is often hard to quantify, but the impact such an application has on a set of targeted effects can be measured using search logs and user surveys. The data collected this way provides an estimate of the value of Findability within the firewall of an organization.

Nowadays, hardly anybody questions the marketing value of a good corporate web site or the impact email has on the way we communicate. Such channels and services are self-evident business enablers today. In this respect, the benefits of precise and quick information access within the corporation should be self-evident. The trick is to get the tool just right.

What Differentiates a Good Search Engine from a Bad One?

That was one of the questions the UIE research group asked themselves when conducting a study of on-site search. One of the things they discovered was that the choice of search engine was not as important as the implementation. Most of the big search vendors were found in both the top sites and the bottom sites.

So even though the choice of vendor influences what functionality you can achieve and the control you have over your content there are other things that matter, maybe even more. Because the best search engine in the world will not work for you unless you configure it properly.

According to Jared Spool there are four kinds of search results:

  • ‘Match relevant results’ – returns the exact thing you were looking for.
  • ‘Zero results’ – no relevant results found.
  • ‘Related results’ – i.e. search for a sweater and also get results for a cardigan. (If you know that a cardigan is a type of sweater you are satisfied. Otherwise you just get frustrated and wonder why you got a result for a cardigan when you searched for a sweater).
  • ‘Wacko results – the results seem to have nothing in common with your query.

So what did the best sites do according to Jared Spool and his colleagues?
They returned match relevant results, and they did not return 0 results for searches.

So how do you achieve that then? We have previously written about the importance of content refinement and information quality. But what do you do when trying to achieve good search results with your search engine? And what if you do not have the time or knowledge to do a proper content tuning process?

Well, the search logs are a good way to start. Start looking at them to identify the 100 most common searches and the results they return. Are they match relevant results? It is also a good idea to look at the searches that return zero results and see if there is anything that can be done to improve those searches as well.

Jared Spool and his colleagues at UIE mostly talk about site search for e-commerce sites. For e-commerce sites bad search results mean loss of revenue while good search results hopefully give an increase in revenue (if other things such as check out do not fail). Working with intranet search the implications are a bit different.

With intranet search solutions the searches can be more complex when information not items, is what users are searching for. It might not be as easy to just add synonyms or group similar items to achieve better search results. I believe that in such a complex information universe, proper content tuning is the key to success. But looking at the search logs is a good way for you to start. And me and my colleagues here at Findwise can always help you how to get the most out of your search solution.