Tag Archives: Google search

Searching for Zebras: Doing More with Less

Posted on by
4

There is a very controversial and highly cited 2006 British Medical Journal (BMJ) article called “Googling for a diagnosis – use of Google as a diagnostic aid: internet based study” which concludes that, for difficult medical diagnostic cases, it is often useful to use Google Search as a tool for finding a diagnosis. Difficult medical cases are often represented by rare diseases, which are diseases with a very low prevalence.

The authors use 26 diagnostic cases published in the New England Journal of Medicine (NEJM) in order to compile a short list of symptoms describing each patient case, and use those keywords as queries for Google. The authors, blinded to the correct disease (a rare diseases in 85% of the cases), select the most ‘prominent’ diagnosis that fits each case. In 58% of the cases they succeed in finding the correct diagnosis.

Several other articles also point to Google as a tool often used by clinicians when searching for medical diagnoses.

But is that so convenient, is that enough, or can this process be easily improved? Indeed, two major advantages for Google are the clinicians’ familiarity with it, and its fresh and extensive index. But how would a vertical search engine with focused and curated content compare to Google when given the task of finding the correct diagnosis for a difficult case?

Well, take an open-source search engine such as Indri, index around 30,000 freely available medical articles describing rare or genetic diseases, use an off-the-shelf retrieval model, and there you have Zebra. In medicine, the term “zebra” is a slang for a surprising diagnosis. In comparison with a search on Google, which often returns results that point to unverified content from blogs or content aggregators, the documents from this vertical search engine are crawled from 10 web resources containing only rare and genetic disease articles, and which are mostly maintained by medical professionals or patient organizations.

Evaluating on a set of 56 queries extracted in a similar manner to the one described above, Zebra easily beats Google. Zebra finds the correct diagnosis in top 20 results in 68% of the cases, while Google succeeds in 32% of them. And this is only the performance of the Zebra with the baseline relevance model — imagine how much more could be done (for example, displaying results as a network of diseases, clustering or even ranking by diseases, or automatic extraction and translation of electronic health record data).

Bridging the Gap Between People and (Enterprise Search) Technology

Posted on by
1

Tony Russell-Rose recently wrote about the changing face of search, a post that summed up the discussion about the future of enterprise search that took part at the recent search solutions conference. This is indeed an interesting topic. My colleague Ludvig also touched on this topic in his recent post where he expressed his disappointment in the lack of visionary presentations at this year’s KMWorld conference.

At our last monthly staff meeting we had a visit from Dick Stenmark, associate professor of Informatics at the Department of Applied IT at Gothenburg University. He spoke about his view on the intranets of the future. One of the things he talked about was the big gap in between the user’s vague representation of her information need (e.g. the search query) and the representation of the documents indexed by the intranet enterprise search engine. If a user has a hard time defining what it is she is looking for it will of course be very hard for the search engine to interpret the query and deliver relevant results. What is needed, according to Dick Stenmark, is a way to bridge the gap between technology (the search engine) and people (the users of the search engine).

As I see it there are two ways you can bridge this gap:

  1. Help users become better searchers
  2. Customize search solutions to fit the needs of different user groups

Helping users become better searchers

I have mentioned this topic in one of my earlier posts. Users are not good at describing which information they are seeking, so it is important that we make sure the search solutions help them do so. Already existing functionalities, such as query completion and related searches, can help users create and use better queries.

Query completion often includes common search terms, but what if we did combine them with the search terms we would have wanted them to search for? This requires that you learn something about your users and their information needs. If you do take the time to learn about this it is possible to create suggestions that will help the user not only spell correctly, but also to create a more specific query. Some search solutions (such as homedepot.com) also uses a sort of query disambiguation, where the user’s search returns not only results, but a list of matching categories (where the user is asked to choose which category of products her search term belongs). This helps the search engine return not only the correct set of results, but also display the most relevant set of facets for that product category. Likewise, Google displays a list of related searches at the bottom of the search results list.

These are some examples of functionalities that can help users become better searchers. If you want to learn some more have a look at Dan Russells presentation linked from my previous post.

Customize search solutions to fit the needs of different user groups

One of the things Dick Stenmark talked about in his presentation for us at Findwise was how different users’ behavior is when it comes to searching for information. Users both have different information needs and also different ways of searching for information. However, when it comes to designing the experience of finding information most companies still try to achieve a one size fits all solution. A public website can maybe get by supporting 90% of its visitors but an intranet that only supports part of the employees is a failure. Still very few companies work with personalizing the search applications for their different user groups. (Some don’t even seem to care that they have different user groups and therefore treat all their users as one and the same.) The search engine needs to know and care more about its’ users in order to deliver better results and a better search experience as a whole. For search to be really useful personalization in some form is a must, and I think and hope we will see more of this in the future.

Quick Website Diagnostics with Search Analytics

Posted on by
Reply

I have recently been giving courses directed to web editors on how to successfully apply search technology on a public web site. One of the things we stress is how to use search analytics as a source of user feedback. Search analytics is like performing a medical checkup. Just as physicians inspect patients in search of maladious symptoms, we want to be able to inspect a website in search of problems hampering user experience. When such symptoms are discovered a reasonable resolution is prescribed.

Search analytics is a vast field but as usual a few tips and tricks will take you a long way. I will describe three basic analysis steps to get you started. Search usage on public websites can be collected and inspected using an array of analytics toolkits, for example Google Analytics.

How many users are using search?

For starters, have a look at how many of your users are actually using search. Obviously having a large portion of users doing so means that search is becoming very important to your business. A simple conclusion stemming from such evidence is that search simply has to work satisfactorily, otherwise a large portion of your users are getting disappointed.

Having many searchers also raises some questions. Are users using search because they want to or because they are forced to, because of tricky site navigation for example? If you feel that the latter seems reasonable you may find that as you improve site navigation your number of searchers will decrease while overall traffic hopefully increases.

Just as with high numbers, low numbers can be ambiguous. Low scores especially coupled with a good amount of overall site traffic may mean that users don’t need search in order to find what they are looking for. On the other hand it may mean that users haven’t found the search box yet, or that the search tool is simply too complicated for the average user.

Aside from the business, knowing how popular search is can be beneficial to you personally. It’s a great feeling to know that you are responsible for one of the most used subsystems of your site. Rub it in the face of your colleague!

From where are searches being initiated?

One of the first recommendations you will get when implementing a search engine for your web site is to include the search box on each and every page, preferably in a standardized easy-to-find place like the top right corner. The point of having the search box available wherever your users happen to be is to enable them to search, typically after they have failed to find what they are looking for through browsing.

Now that we know that search is being conducted everywhere, we should be keeping an eye out for pages that frequently emit searches. Knowing what those pages are will let us improve the user experience by altering or completing the information there.

Which are the most common queries?

The most frequently issued queries to a search system make up a significant amount of the total number of served queries. These are known as head queries. By improving the quality of search for head queries you can offer a better search experience to a large amount of users.

A simple but effective way of working with search tuning is this. For each of the 10, 20 or 50 most frequent queries to the system:

  1. Imagine what the user was looking for when typing that query
  2. Perform that query yourself
  3. Examine the 5-10 top results in the result list:
    • Do you think that the user was content with those results
    • If yes, pat your back 🙂
    • If not, tweak using synonyms or best bets.

Go through this at least once a month. If the information on your site is static you might not need to change a lot of things every time, but if your content is changing or the behavior of the users you may need to adjust a few things.

Systematic Relevance: Evaluation

Posted on by
Reply

Perfect relevance is the holy grail of Search. If possible we would like to give every user the document or piece of information they are looking for. Unfortunately, our chances of doing so are slim. Not even Google, the great librarian of our age, manages to do so. Google is good but not perfect.

Nevertheless, as IT professionals, search experts and information architects we try. We construct complicated document processing pipelines in order to tidy up our data and to extract new metadata. We experiment endlessly with stop words, synonym expansion, best bets and different ways to weigh sources and fields. Are we getting any closer? Well, probably. But how can we know?

There are a myriad of knobs and dials for tuning in an enterprise search engine. This fact alone should convince us that we need a systematic approach to dealing with relevance; with so many parameters to work with the risk of breaking relevance seems at least as great as the chance of improving on it. Another reason is that relevance doesn’t age gracefully, and even if we do manage to find a configuration that we feel is decent it will probably need to be reworked in a few months time. At Lucene Eurocon Grant Ingersoll also said that:

“I urge you to be empirical when working with relevance”

I favor the trial and error approach to most things in life, relevance tuning included. Borrowing concepts from information retrieval, one usually starts off by creating a gold standard. A gold standard is a depiction of the world as it should be: a list of queries, preferably popular or otherwise important, and the documents that should be present in the result list for each of those queries. If the search engine were capable of perfect relevance then the results would be 100% accuracy when compared to the gold standard.

The process of creating such a gold standard is an art in itself. I suggest choosing 50 or so queries. You may already have an idea of which ones are interesting to your system; otherwise search analytics can provide this information for you. Furthermore, you need to decide which documents should be shown for each of the queries. Since users are usually only content if their document is among the top 3 or 5 hits in the result list, you should have up to this amount of documents for each query in your gold standard. You can select these documents yourself if you like. However, arguably the best way is to sit down with a focus group selected from among your target audience and have them decide which documents to include. Ideally you want a gold standard that is representative for the queries that your users are issuing. Any improvements achieved through tuning should boost the overall relevance of the search engine and not just for the queries we picked out.

The next step is to determine a baseline. The baseline is our starting point, that is, how well the search engine compares out of the box to the gold standard. In most cases this will be significantly below 100%. As we proceed to tune the search engine its accuracy, as compared to the gold standard, should move from the baseline toward 100%. Should we end up with accuracy below that of the baseline then our work has probably had little effect. Either relevance was as good as it gets using the default settings of the search engine, or, more likely, we haven’t been turning the right knobs.

Using a systematic approach like the one above greatly simplifies the process of working with relevance. It allows us to determine which tweaks are helpful and keeps us on track toward our ultimate goal: perfect relevance. A goal that, although unattainable, is well worth striving toward.

Six Simple Steps to Superior Search

Posted on by
Reply

Do you have your search application up and running but it still doesn’t quite seem to do the trick? Here are six simple steps to boost the search experience.

Avoid the Garbage in-Garbage out Syndrome

Fact 1: A search application is only as good as the content it makes findable

If you have a news search service that only provides yesterday’s news, the search bit does not add any value to your offering.

If your Intranet search service provides access to a catalog of employee competencies, but this catalog does not cover all co-workers or contain updated contact details, then search is not the means it should be to help users get in touch with the right people.

If your search service gives access to a lot of different versions of the same document and there is no metadata available as to single out which copy is the official one, then users might end up spending unnecessary time reviewing irrelevant search results. And still you cannot rule out the risk that they end up using old or even flawed versions of documents.

The key learning here is that there is no plug and play when it comes to accurate and well thought out information access. Sure, you can make everything findable by default. But you will annoy your users while doing so unless you take a moment and review your data.

Focus on Frequent Queries

Fact 2: Users tend to search for the same things over and over again.

It is not unusual that 20 % of the full query volume is made up of less than 1 % of all query strings. In other words, people tend to use search for a rather fixed set of simple information access tasks over and over again. Typical tasks include finding the front page of a site or application on the Intranet, finding the lunch menu at the company canteen or finding the telephone number to the company helpdesk.

In other words, you will be much advised to make sure your search application works for these highly frequent (often naïve) information access tasks. An efficient way of doing so is to keep an analytic eye on the log file of your search application and take appropriate action on frequent queries that do not return any results whatsoever or return weird or unexpected results.

The key learning here is that you should focus on providing relevant results for frequent queries. This is the least expensive way to get boosted benefit from your search application.

Make the Information People Often Need Searchable

Fact 3: Users do not know what information is available through search.

Users often believe that a search application gives them access to information that really isn’t available through search. Say your users are frequently searching for ”lunch menu”, ”canteen” and ”today’s lunch”, what do you do if you do not have the menu available at all on your Intranet or Web site?

In the best of worlds, you will make frequently requested information available through search. In other words, you would add the lunch menu to your site and make it searchable. If that is not an option, you might consider informing your users that the lunch menu—or some other popular information people tend to request—is not available in the search application and provide them with a hard-coded link to the canteen contractor or some other related service as a so called “best bet” (or sponsored link as in Google web search).

The key learning here is to monitor what users frequently search for and make sure the search application can tackle user expectations properly.

Adapt to the User’s Language

Fact 4: Users do not know your company jargon.

People describe things using different words. Users are regularly searching for terms which are synonymous to—but not the same as—the terms used in the content being searched. Say your users are frequently looking for a ”travel expense form” on your Intranet search service, but the term used in your official company jargon  is ”travel expenses template”. In cases like this you can build a glossary of synonyms mapping those common language terms people tend to search for frequently to official company terms in order to satisfy your users’ frequent information needs better without having to deviate from company terminology. Another way of handling the problem is to provide hand-crafted best bets (or sponsored links as in Google web search) that are triggered by certain common search terms.

Furthermore, research suggests that Intranet searches often contain company-specific abbreviations. A study of the query log of a search installation at one of Findwise’s customers showed that abbreviations—query strings consisting of two, three or four letters—stood for as much as 18 % of all queries. In other words, it might be worthwhile for the search application to add the spelled-out form to a query for a frequently used abbreviation. Users searching for “cp” on the Intranet would for example in effect see the results of the query “cp OR collaboration portal”

The lesson to learn here is that you should use your query log to learn the terminology the users are using and adapt the search application accordingly, not the other way around!

Help Users With Spelling

Fact 5: Users do not know how to spell.

Users make spelling mistakes—lots of them. Research suggests that 10—25 % of all queries sent to a search engine contain spelling mistakes. So turn on spellchecking in your search platform if you haven’t already! And while you are at it, make sure your search platform can handle queries containing inflected forms (e.g. “menu”, “menus”, “menu’s”, “menus’”). There’s your quick wins to boost the search experience.

Keep Your Search Solution Up-To-Date

Fact 6: Your search application requires maintenance.

Information sources change, so should your search application. There is a fairly widespread misconception that a search application will maintain itself once you’ve got it up and running. The truth is you need to monitor and maintain your search solution as any other business-critical IT application.

A real-life example is a fairly large enterprise that decided to perform a total makeover of its internal communication process, shifting focus from the old Intranet, which was built on a web content management system, in favor of a more “Enterprise 2.0 approach” using a collaboration platform for active projects and daily communication and a document management system for closed projects and archived information.

The shift had many advantages, but it was a disaster for the Enterprise Search application that was only monitoring the old Intranet being phased out. Employees looking for information using the search tool would in other words only find outdated information.

The lesson to learn here is that the fairly large investment in efficient Findability requires maintenance in order for the search application to meet the requirements posed on it now and in the future.

References

100 Most Often Mispelled Misspelled Words in English – http://www.yourdictionary.com/library/misspelled.html

Definition of “sponsored link” – http://encyclopedia2.thefreedictionary.com/Sponsored+link

Interesting New Search Features

Posted on by
Reply

Out on the web there are a large number of small search engines that try to stand out and maybe take some of the market shares from Google. Many of them have interesting search features.

I would like to introduce some of them in order to help other realize that search can (and should) be a bit more then a search bar and a list of hits. A number of these alternative search engines have focused on the visual presentation of the search result in interesting ways. For example the search engine quintura uses tag clouds of related terms and concepts to the original query.

A slightly different approach has been taken by mnemomap and webbrain that presents related concepts in a graph instead. The other part is to visually show the divisions of the search results into different categories so they can easily be navigated through but also to give a quick overview of the subject, examples of that can be seen at e.g. mooter and kooltorch. Finally I would also like to mention kartOO that have, in my opinion, gone one step further and even presents the links to the search results with images and icons.

In conclusion one can say that the ability to graphically visualize the search result so that it is possible to get a quick overview of a particular subject can prove to be a very important feature in future search solutions. It would not only help users find what they want to know, but also help them get a better and wider understanding of a particular subject, without forcing them to read through a large chunk of (hopefully) relevant text.

The search result and related concepts can be presented graphically instead. That will also take advantage of the fact that people can take in a lot more information through an image then by reading text. Further it can help the user to easily see if he or she is on the right track and make possible refinements to the query even before any returned document has been read through, thus saving valuable time, which today is more important then ever.